Make Catalog Candidate Review Repeatable Before Any Production Mutation

Flow

Fail-Closed Candidate Planning

Each guard either narrows the evidence or stops before a fact can be promoted.

1. 1Require safe mode⌄

   Demand metadata-only, chunking-enabled, no-persistence flags.

2. 2Bind exact bytes⌄

   Verify the retained payload against its expected hash.

3. 3Reject malformed input⌄

   Report the failure without logging raw payload values.

4. 4Filter identities⌄

   Flag missing or duplicate stable identifiers.

5. 5Canonicalize candidates⌄

   Hash stable representations without declaring business truth.

6. 6Plan chunks⌄

   Create deterministic groups and group fingerprints.

7. 7Emit receipt⌄

   Return counts, identities, and explicit zero-write assertions.

Large catalog payloads create pressure to move quickly.

A system fetches a product feed, parses a large list of records, and the natural next step seems obvious: normalize the records and write them into production.

That is the dangerous shortcut.

A fetched payload is not canonical business truth. It is evidence.

The safer pipeline treats retained payloads as the input to a deterministic review plan, not as immediate write authority.

Illustrative anonymized example
Retained evidence
Hash verification
In-memory candidate normalization
Deterministic chunk planning
Reviewable candidate ledger
Separate approval for production mutation

The public-safe thesis is:

A data integration pipeline can make catalog review safer by chunking retained evidence deterministically before any normalization or production mutation path runs.

This is not a claim that deterministic chunking proves data quality. It does not. It only makes candidate review repeatable.

Evidence is not fact

A retained payload tells the system:

• This is what arrived from the upstream source.
• This is the payload reference.
• This is the expected content hash.
• This is the byte sequence used for review.

It should not automatically tell the system:

• These are canonical product records.
• These records should update production.
• These values should overwrite existing facts.

That boundary matters because upstream catalog data can be malformed, stale, duplicated, incomplete, or simply not approved yet.

The retained payload is useful because it freezes the evidence. Review does not need to re-fetch from the upstream system, where the data might have changed.

The chunk planner creates stable review units

A catalog payload can be too large to review as one object. The chunk planner turns candidate rows into smaller units.

A chunk plan might include:

Illustrative anonymized example
{
  "outcome": "catalog_chunked_in_memory",
  "candidateCount": 1201,
  "chunkCount": 3,
  "deterministicOrderingStrategy": "sort by stable source identity, then content hash, then chunk by maxPerBatch",
  "persisted": false,
  "dbWrites": false,
  "canonicalFactsCreated": false,
  "rawValuesPrinted": false
}

The important fields are not only the counts. The mutation flags matter.

They say:

• This was a review plan.
• It did not write database rows.
• It did not create source observations.
• It did not create canonical facts.
• It did not print raw values.

A safe catalog review path should be able to prove that distinction.

Determinism makes review reproducible

A deterministic chunk plan should produce the same chunks from the same retained payload.

That requires stable inputs and stable ordering.

For example:

1. Resolve retained payload by reference.
2. Verify payload hash.
3. Parse candidate rows in memory.
4. Keep only rows with a stable source identity.
5. Hash row content using stable JSON.
6. Sort by identity, then content hash.
7. Split into fixed-size chunks.
8. Return chunk metadata only.

If the upstream payload contains the same records in a different row order, the chunk plan should remain stable.

That is useful for review, audit, and retry behavior.

The reviewer can ask:

• Am I reviewing the same candidate set as before?
• Did this plan come from the same retained payload?
• Did the chunk hash change?
• Were any writes performed?

The chunk plan should answer those questions.

The wrapper is part of the contract

A safe wrapper should require explicit flags:

Illustrative anonymized example
{
  "metadataOnly": true,
  "allowCatalogCandidateChunking": true,
  "noPersistence": true
}

These flags are not decoration. They are the boundary.

The system should reject a chunk planning request unless it is explicitly in metadata-only, chunking-allowed, no-persistence mode.

That keeps the review path separate from the mutation path.

Failure modes

| Failure | Detection | Safe result | |---|---|---| | Payload hash mismatch | Compare retained hash to expected hash | Stop before parsing | | Malformed JSON | Parse failure | Report malformed without raw body | | Operational truth field appears | Field allow/block list | Block normalization | | Duplicate identity appears | Stable identity map | Exclude or flag duplicate candidates | | Chunk plan mutates state | Static tests and output flags | Fail test |

The useful system property is repeatability, not automatic correctness.

A deterministic chunk plan does not certify the catalog. It gives the team a stable review surface before production facts are created.

// Illustrative anonymized example
function batchIdCandidate(batchHash: string) {
  return `batch_${batchHash.replace(/^sha256:/, '').slice(0, 32)}`
}