Flow
When the write is interrupted or rerun
Taken from the write step's tests.
1Check the copy
The fingerprint is verified before anything is written.
2Write the run record
One record for the whole import goes first.
3Write batches in order
Each result says new, replayed, already received, or conflict.
4A batch fails
The writer stops there. Earlier batches stay recorded.
5Run it again
Batches already saved come back as repeats, so nothing is counted twice.
bar chart
Response size against the two limits (illustrative)
Illustrative relative scale, not measured values. The response sat above the probe cap and below the policy ceiling, which is why the probe refused and the capture succeeded.
The first time the ordering system asked the stores' sales feed for its product list, my probe read the size header and hung up. The feed said it was several times bigger than the limit I'd set. So the probe cancelled, read nothing, and said "too big."
That's what I wanted. If you only check size while reading, you've already started a download you might not finish.
No way to ask for less
The vendor's feed has a "changed since" filter and an "active only" flag, but no way to ask for one page at a time. My own gateway in front of it didn't pass limits along either. A second probe confirmed it without reading any data: this feed is all or nothing.
So I accepted one big read, on purpose, with rules.
Read it once, and lock it up
The rules: a hard ceiling above the real size, a short shelf life for the raw copy, compression, encryption, and proof the key is backed up before any fetch.
The capture step reads the whole thing once and checks size twice, once from the header and once by counting bytes as they arrive. Count the bytes even when the header looks fine.
It adds up the size of each chunk as it arrives and reports too big as soon as the total passes the limit.
Then it fingerprints the exact bytes with a hash, encrypts them, stores them, and hands back a reference. It doesn't parse anything.
Every step checks the fingerprint
Every later step starts from that reference. It unlocks the copy, checks the fingerprint, and only then reads the JSON. If the fingerprint is wrong, it stops.
Identity comes from ID fields only, never names. Exact duplicate rows collapse. Rows that share an ID but disagree all get thrown out. In my real catalog, all of the roughly ten thousand rows had good IDs and none were duplicates.
Rows get sorted and cut into fixed batches of a few hundred. Each batch gets a fingerprint, and that fingerprint plus the run goes into its idempotency key, a label that lets the receiver spot a repeat. Row IDs no longer depend on where the copy is stored.
Preview, then write
A dry run builds every command and validates it without sending. The real write rebuilds the same plan, then sends the run record and the batches. Each result says new, replayed, already received, or conflict. If a batch fails, it stops there. If you run it again, nothing gets counted twice.
Everything landed as source evidence. No real products, mappings, or stock numbers were created. Matching those rows to real products is the next piece of work.