Back to Feed
article 3m read

Telemetry sent its strangest events with the least information

The hook collector built a thin payload whenever it couldn't read my coding agent's version, which hid the events I'd most want to inspect. Writing unknown as a value fixed the shape, but the same cleanup left the server never checking the version at all.

Article DepthPractice

Flow

The version check I plan to add

A plan, not shipped code.

  1. 1Version arrives

    Read from the envelope; the envelope shape no longer depends on it.

  2. 2Known range

    Mark the row supported and process it normally.

  3. 3Newer than checked

    Process it, mark it unverified, and count it.

  4. 4Not readable

    Process it, mark it unknown, and count it separately.

  5. 5Reject later, maybe

    Only after the counts have been watched for a while.

Somewhere in the telemetry collector sat a rule I wrote and then stopped reading: if it couldn't find my coding agent's version, it sent less, and the server agreed with it.

From hook to stored row

My agent fires hooks when something happens, like a prompt being submitted or a tool call finishing. Each hook runs a small script that hands the event to my own command-line tool, which wraps it in an envelope, sends it to my ingest server, and logs any delivery error instead of failing. A comment says it also passes the original input back to the agent, but I haven't confirmed the code does that. The server validates the envelope, picks a processor, and stores one row.

The branch that decided the shape

The collector checked one thing: could it find a version in the environment. If it could, it sent the full envelope with the tool name, its version and the schema. If it couldn't, it sent a thin event with none of those.

One check on the version picked between a full event and a thin one.

The events without a version are the unusual ones, like a stripped environment, a new build or a configuration I hadn't seen before. I'd bet those are the ones most worth investigating, and they arrived with the least information, so I couldn't even count them.

The server agreed with it

The server had several processors. Some were placeholders that only threw errors, and a legacy processor took anything marked "unknown" plus anything no other processor wanted.

One envelope, every time

Now the collector always sends the full envelope and writes "unknown" wherever it can't read a value, which gives me something I can filter and count. The source label also stopped carrying the version, so my charts don't grow a new group with every release.

What the registry stopped asking

I also cut the server down to a single processor, and now it never looks at the version at all. It stores the value and moves on. If the agent renames a field, I expect the row would still save, just with empty columns, although I haven't tested that. My plan is to mark every row as supported, newer than anything I've checked, or unknown, and to count those before I ever reject anything. It isn't built yet.

Two smaller versions of the same mistake

Any hook name I haven't mapped becomes a generic notification, and a separate list guards the route, so if those two lists drift apart, unusual events will pass as ordinary ones. Also, a tool call without an error message counts as a success, even if it failed without reporting anything.

The catch block I still need to read

A later commit made the error path dump the entire error object to stderr. That request can carry my auth token, and HTTP client errors often keep the request details attached. I haven't checked the output yet. Once I have, the dump gets replaced by a few safe fields: hook type, payload size, status code and a short message.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Related idea

Auth middleware would have logged everyone out the minute the auth server hiccupped

The SDK middleware turned every failed token check into a redirect to sign-in, so an outage on my side would have looked like a mass logout. I split the answer into yes, no and couldn't-check, and made the middleware answer an outage with a 503. In the same commit the browser client went the other way and now throws away good tokens when the server is unreachable.

via #reliability · #contracts