A hard cap on an import, so the lookup got sliced and the slices made to agree

Flow

What the merge does with each kind of field

1. 1Whole-question fields⌄

   Must be the same in every slice, or the import fails.

2. 2Product descriptions⌄

   Must match exactly when a product shows up twice.

3. 3How it was found⌄

   Combined from all slices.

4. 4Totals and verified flag⌄

   Worked out again from the merged result.

The vendor I was onboarding was too big for my own import. The ordering system had a rule: if it saw more than a few hundred product sightings, it stopped. A sighting is a store plus that store's product ID.

The rule was there because the import sends all its sightings to a catalog service in one request. The service sends back one answer: which products exist, where each is sold, what it couldn't match, and how complete the answer is. My import capped how big that request could be. I assume the service has a limit too.

But my rule was bad. It counted the sightings before duplicates were removed, not the clean list it actually sent. And it just failed. A retry would fail the same way, every time.

Cutting the question up

I split the code that builds the request from the code that sends it. Now the builder makes two clean lists with no repeats, one for stock records and one for sales. Then a loop cuts both lists into slices the size of the old limit and sends them one at a time. The longer list decides how many calls to make, and there's always at least one.

I kept a bigger total limit, in the thousands, counted on unique items. It's checked before the first call. If the input is too big, it fails at once and costs nothing.

Putting the answers back together

This is the part that can lie to you. One answer is trustworthy because the service made it all at once. Two answers might not match. One slice might say a store was fine while another says it was having trouble. If you just glue them, you report a world that never existed.

So the merge has a rule for each kind of field.

• Fields about the whole question, like which stores and how they were doing, must be the same in every slice.
• A product that shows up twice must be described the same both times, including where it's sold.
• Evidence of how a product was found gets combined.
• Totals and the "verified" flag get worked out again from the merged result.

If anything that must match doesn't, the import fails. It doesn't guess. Things the service couldn't match get passed along, not thrown away.

The core check is small.

If any slice's scope differs from the first slice's, the merge throws an error.

The merged answer also gets its own new fingerprint, so nobody mistakes it for a single answer. If only one slice was needed, the answer comes back as is.

Cases against a fake service

I wrote a few tests against a fake service. Just over the old limit now works as one full slice and a small one. Over the new limit fails with no calls. Two slices that found one product in different ways merge cleanly. Two slices that disagree on where a product is sold fail.

What I don't know is how often real slices will disagree. The calls go one at a time, which is slow, and the catalog can change during the run. Then the import fails and someone reruns it. The weak spot is that the list of fields to compare is written by hand, so a new field nobody adds to it gets ignored.