Flow
My order for cleaning it up
1Tag every row
Each run and result says live or simulated.
2Count only live
The run limit and every list skip simulated rows by default.
3Missing key is an error
The agent stops with a clear message and writes nothing.
4No name-based demo
Demo is a setting a person chose, never a repo name.
5Honest stubs
Unfinished routes say not built yet instead of success.
6Check filenames
Plain names with a markdown ending, inside the folder.
The demo switch I shipped doesn't demo anything. Turn it on and start a run. A few seconds later a new result sits in the table for real results. The run is marked done, and the account has one less run left.
The feature is a good one. I added an agent to the admin console of a developer-tools project I work on. It reads a repo and writes "skills", short notes that tell other coding agents how the code works. It all went in as one big commit with no tests. Reading it back, I found three ways it pretends and one loose spot in how it saves files.
The row that says a run finished
With demo on, the launch route skips the sandbox. It still makes the same records a real run makes. Then it waits a few seconds, saves a fake tip, and marks the run done. No field says "demo".
The fake tip is a high-priority, near-certain note about login security, on whatever repo you picked. It points at a file that is never written. And the run limit counts done runs, so each demo uses up a real one. A few demos would burn a beta account's whole allowance.
One sample repo name also forces demo mode, and the demo badge in the settings won't show it.
The key that isn't there
The agent has its own pretend path. If its model key is missing, it doesn't stop. It writes two placeholder skills into the real skills folder. Then it says action should be taken, at half confidence. Half is still a number something can act on.
Other agents are meant to read that folder. A made-up note telling an agent how to start an app is not harmless once something follows it.
Two routes that say yes
One new route says a job was queued and queues nothing. Another always returns an empty history, because its query is commented out. Both look like normal answers.
The filenames come from the model
The agent saves files under whatever names the model sends back. A name that climbs out of the folder writes somewhere else. Only accept a plain name with a markdown ending, and check that the final path is still inside the folder.
Where I'd start
None of this is fixed in this commit. The first thing I'd add is a tag on every run and result that says live or simulated. Every list would show live rows only by default.
The run limit would then count only completed live runs.
After that, a missing key is an error, not a switch to fake output. The sample-name override goes. The stub routes say "not built yet" until they're built. Then the filename check.