Flow
Kill a run first, then pick a design
1Start a run
The run says running and the upload is open.
2Kill the process halfway
Stands in for a time limit, a deploy or running out of memory.
3Try to read the log
With one big upload, expect nothing readable.
4Choose a design
Small log pieces, a last-seen time, and a sweeper for quiet runs.
5Kill it again
Expect most of the log readable and the run marked failed.
The agent runner has a comment that says its logs survive a crash. I wrote that in the same change that moved the run into the background, and the two don't fit together.
Moving the run behind the response
The runner puts a coding agent in a throwaway sandbox with a repository. The agent writes a few instruction files, and the runner opens a pull request with them. A run takes minutes, and the web request used to wait the whole time. Now the request saves two rows marked running, returns their IDs, and starts the real work after the reply is sent.
Where the log actually lives
With no one waiting, the log is the only record of what the agent did. So the runner opens an upload to object storage before the agent starts and writes each chunk of output into it. After the agent ends, the runner parses its output and opens the pull request, and only then closes the upload. On an error it closes the upload too. Either way it saves its address on the run.
The catch is that this kind of upload isn't a file until it's closed. Until then, nobody can read it. The upload tool also holds data in memory until it has a few megabytes, and a whole run might not produce that much. So during a run, the log mostly sits in memory.
If the process gets killed, by a time limit, a deploy or running out of memory, the close never runs. There's no log and no address, and the run says running for good, as far as I can tell. The new runs page will keep showing it as Running. Streaming only helps with errors my code can catch, and those were never the scary ones.
I'd look at the background time limit first, since I raised it in this same change and runs are long. Nothing in the commit tests what happens there.
The parser expects one event per line
The runner now expects one JSON event per line and looks for the last result event to get cost, tokens, time and the final answer. That beats the old guessing, if it works. The agent change in the same commit prints each event spread over many lines, and I didn't check that the runner can read that. It still pulls the decision out of text, and it still finds created files by matching paths in the log.
Two bugs in the run cap
The run cap counts only finished runs, so runs started at the same moment all pass. And the setting that raises the cap only reads its last digit. Neither is fixed yet.
Small numbered log pieces and a sweeper
If you're building something like this, these are what I'd want:
- Write the log in small pieces, each one its own file, every few seconds.
- Save a last-seen time on the run with every piece.
- Run a sweeper that marks quiet runs as failed, with a wait longer than the agent's longest pause.
Each piece is written as its own numbered file.
Each piece is readable the second it lands, so a kill only loses the last few seconds. I haven't built this yet.
If I did it again, I'd kill a run halfway on purpose before writing that comment.