Flow
Each drop and the field that reports it
Five drops, five counters in the response.
1Too many rows
Report whether the limit was hit.
2Narrow rules
Report which rule groups ran.
3Too many matches
Count the matches that were dropped.
4Long lists
Flag lists that were cut.
5Errors
Count failed messages and mark the scan partial.
Flow
What I'd change first
Nothing here is built yet; this is my order, not a plan with dates.
1Full secret in the response
Send only the masked value.
2Failures counted as clean
Count them and mark the scan partial.
3Unmasked preview
Mask it, and repeat the organization filter.
4Loose ID-number rule
Lower its confidence or tighten the match.
5No tests
Write tests that hit each limit on purpose.
The scanner page I added today can count the leaked keys it finds. It can't tell me what it skipped. After reading the diff, the list of skips is longer than the list of checks.
The page reads one organization's saved agent chats and scans each message for secrets and personal data: keys, connection strings, card numbers, ID-shaped numbers, emails and IP addresses. It went in as two commits on the same day. The first used outside tools. The second swapped them for a local set of patterns, and the prompt-injection check was dropped along the way.
It checks scope and cleans inputs
It checks scope first. The route finds the caller's organization in the app's own database. If there's no record, it refuses before reading anything. The first version would have scanned the caller's own rows instead. For a page that reads other people's chats, a refusal is what I want.
It also cleans up its inputs. Junk in the day window or row limit used to turn into not-a-number and go straight to the database. Now it falls back to a default and stays inside a floor and a ceiling.
Where findings go to disappear
After that, the scan drops things at five points, and the result never says so. If you build something like this, each drop needs its own counter in the response.
Too many rows. It reads the newest messages first and stops at a fixed count. A busy organization only gets its newest slice checked. Report whether the limit was hit.
Narrow rules. Only the high-confidence secret patterns run, and broken patterns are skipped. The result still says every scanner was on. Report which rule groups actually ran.
Too many matches. Each secret pattern stops after a few hits in one message, on the theory that so many hits must be noise. A pasted file full of real keys looks just like that. Count what got dropped.
Long lists. Each message's findings are sorted by severity and cut at a limit. Sorting first is right, but a cut list looks complete. Flag the cut.
Errors. When the scanner fails on a message, it returns an empty result. The route counts that message as scanned and clean. Count failures, and mark the whole scan partial when there are any.
Loose rules, an unmasked preview, no tests
All personal-data hits are labeled high confidence. Card numbers get a real checksum. The ID-number rule matches almost any nine digits, yet it's marked critical. Probably a lot of those hits are order numbers or timestamps.
Each finding also shows the first thing the user said in that session, unmasked. That lookup doesn't repeat the organization filter. It's probably fine, since the session IDs came from the scoped query, but I'd rather the filter be there. The preview can also show the very secret the finding hides.
There are no tests yet. Everything here comes from reading code.
The full secret also travels in every response, next to the masked one. Of everything on this list, that's what I'd change first, then the failure count, then the unmasked preview.