The Customer Import That Didn’t Need to Check Every Record

Importing customers sounds like a solved problem until the import runs every night and the CRM charges you for every question you ask it. This is a walkthrough of a customer import that checked every record individually, and how a careful read of the API documentation changed the plan.

The scenario

A services company syncs its customer list from its billing system into a hosted CRM. A Laravel command runs nightly. For each of roughly 8,000 customers, it does two things: it asks the CRM whether the customer already exists, and then either creates or updates the record. That is two requests per customer, around 16,000 calls a night, on a platform that bills by request and enforces a daily quota.

The import had started to hit its quota on busy days. The first instinct was to ask for a bigger plan. A better question was: do we actually need to make this many calls?

Step one: measure

We added simple counters to the command and ran it for several nights. What we learned:

• About 94% of the "does this exist?" checks returned an existing customer.
• Only around 3% of customers had changed since the previous run, yet every one was sent as an update.
• The run took more than an hour, almost entirely network wait.

So most of our calls were asking questions we already knew the answer to, and sending changes that were not changes.

Step two: read the documentation

Reading the API reference carefully turned up several things the original integration never used.

An upsert endpoint

The CRM provides a batch upsert that accepts up to 100 contacts, matched on an external ID or email, and creates or updates each as needed. That removes the existence check entirely, so the two calls per customer become one call per 100 customers. For 8,000 customers that is 80 requests instead of 16,000, a reduction of about 99.5%.

A modified-since filter

Our billing system records an updated_at timestamp. If we only send customers changed since the last successful run, the nightly payload drops from 8,000 records to a few hundred. Combined with batching, a typical night could take just a handful of calls.

A claim we cannot confirm

Our code sent a dedupe=true flag on every request. It is not in the documented parameters. It may have been left over from an older API version, or the vendor may honour it without documenting it. We have marked it unverified. Until it is tested, we treat deduplication as our own responsibility rather than trusting a flag we cannot find in the spec.

A sketch of the new import

$since = Cache::get('crm.last_sync', now()->subDay());

Customer::where('updated_at', '>=', $since)
    ->orderBy('id')
    ->chunkById(100, function ($customers) {
        $payload = $customers->map(fn ($c) => [
            'external_id' => $c->id,
            'email'       => $c->email,
            'name'        => $c->name,
        ])->all();

        Http::withToken(config('services.crm.token'))
            ->post(config('services.crm.url') . '/contacts/upsert', [
                'contacts' => $payload,
            ])
            ->throw();
    });

Cache::put('crm.last_sync', now());

Notice that the sync timestamp is saved only after the loop completes. If the job fails halfway, the next run starts from the old timestamp and nothing is lost. Note also that an upsert is safe to repeat, which makes retries much less scary.

The plan, written down

We did not change anything yet. First we wrote a one-page plan with the expected savings, the risks, and the open questions, and asked the team to agree on it. The risks we listed:

• Matching rules: if the upsert matches on email and two customers share one, records could merge. We need to confirm what the CRM does on conflict.
• Deleted customers: a modified-since filter never sees deletions. We need a separate, occasional full reconciliation.
• Clock drift: using "last run time" can miss records saved during the run. We subtract a small safety margin and rely on the upsert being idempotent.
• Partial batch errors: we must read the per-item results rather than trusting a 200 status.

What is still unverified

We have not tested the upsert's behaviour on duplicate emails, the real size limit under our payloads, or whether dedupe=true has any effect. The headline saving is accurate for the number of calls. Whether it saves money depends on how the vendor counts items inside a batch, and we have not confirmed that yet.

How we will roll it out

1. Test the upsert in the CRM sandbox, including duplicate and invalid records.
2. Run the new import in dry-run mode alongside the old one and compare the resulting contacts.
3. Schedule a weekly full reconciliation to catch deletions and missed changes.
4. Switch over, keep the old command for two weeks, then remove it.

Takeaway

The best optimisation here was not technical cleverness. It was asking whether each request needed to exist. Measure what your job really does, read the documentation for the features you are not using, and be honest in writing about which claims you have not tested yet. That habit saves money, and it also saves you from confident mistakes.