Evaluate Evidence Before Letting Automation Act

Flow

A practical build and verification loop

1. 1Contract⌄

   Bound evidence, claims, outcomes, scopes, and digests.

2. 2Adapter⌄

   Enforce timeout, cancellation, and safe provider parsing.

3. 3Store⌄

   Insert immutably and verify read-back.

4. 4Rollout⌄

   Off to shadow to advisory, with authority unchanged.

If you are adding an evidence evaluator to a working product, treat it as a new read path first. The implementation should be useful when the evaluator is unavailable, and it should be impossible for an evaluator response to sneak around the existing write boundary.

Start with a narrow contract. Define an evidence envelope with an opaque evidence ID, a scope ID, a canonical byte representation, a SHA-256 digest, and a bounded list of excerpts. Define a claim envelope with a claim ID, a plain-language question, and the policy or screen that requested it. Define an assessment with an outcome such as supported, unsupported, uncertain, invalid-input, or unavailable. Keep provider names and raw provider errors out of the public result unless there is a deliberate redaction policy.

A service boundary can look like this in pseudocode:

prepare = prepareEvaluation(claim, evidence) if prepare.mode is off: return not_evaluated request = buildProviderRequest(prepare.safeExcerpt, prepare.claim) raw = provider.evaluate(request, timeout) assessment = parseAndValidate(raw) stored = store.insertImmutable(assessment, prepare.digest) if store.read(stored.id).digest != prepare.digest: return persistence_failed return stored

The important details are the guards around the happy path. The preparation step must verify scope and refuse an empty or oversized excerpt. It should select relevant evidence without obeying text inside the excerpt. The provider adapter should enforce a timeout and cancellation signal. The parser should accept only a small schema. The store should use an insert-only record or an append-only revision so a later retry cannot silently rewrite history. The final read-back turns “the database call returned” into the stronger statement “the record is retrievable with the same identity.”

Build a test matrix before adding a UI. Include a valid supported result; a valid uncertain result; malformed JSON; an unknown outcome; an overlong explanation; provider timeout; cancellation; a provider error containing a secret; a duplicate request; a changed excerpt; a scope mismatch; a failed insert; and a successful insert followed by failed read-back. For each case, assert both the returned status and whether a durable record exists. Do not only test the sentence shown to a reviewer.

Use a mode flag with an explicit default. Off means no provider call. Shadow means the provider may be called and the result stored, but no existing behavior reads it. Advisory means a reviewer can see the result beside the existing evidence. Avoid a vague “enabled” flag whose effect changes by deployment. Log mode, latency bucket, outcome class, and failure reason without logging raw evidence or provider payloads.

Roll out in stages. First prove that the evaluator can be removed without changing the product’s authoritative path. Next run shadow evaluation on representative fixtures. Compare reviewer decisions to the typed outcomes, but do not call agreement a proof of truth. Then add an advisory panel with a link to the evidence digest and a clear “not authoritative” label. If the evaluator increases confusion, storage cost, or privacy risk, turn it off while retaining the evidence needed to understand why.

The definition of done is not “the model returned a good answer.” It is: the input is bounded; the claim is explicit; the answer is typed; failure is visible; the record is durable and replayable; scopes are enforced; secrets are absent from logs; and no evaluator result has a mutation path. That is the difference between adding an opinion and adding a controllable system component.