Flow
Each gap and the test that checks it
1The list gets read
Test: one allowed program gets through, an interpreter is refused.
2A failure says no
Test: a failing command reports failure and its exit code.
3A search word stays a word
Test: a word starting with a dash is found.
4The fence follows the project
Test: a file inside the project is readable, a system file outside is not.
The tool runtime under my coding model was telling it that failed commands had worked.
I read one cleanup change to that runtime and its tests. I didn't test the model itself, so what I say about its reactions is a guess.
What the model can see
The coding model I'm building can't touch the machine. It asks for an action, and a runtime checks whether that's allowed, runs it, and hands back two things: some text and a yes or no. That's all the model ever learns about what happened.
A yes that meant "it started"
The shell runner said yes whenever the command started. A command that failed still came back as a yes. If it failed without printing anything, the model got a line with the exit code, still marked yes. Warnings from a command that worked were thrown away.
Likely the model trusts the yes more than the text, and then plans its next step on something that never happened. A crash would have been kinder.
The yes is now earned.
The success flag is now true only when the exit code is zero.
A list nobody read
The config has a spot for the shell programs the model may run. The part that approves actions only checked file paths and never read that list. Now it splits the command the way a shell would, refuses anything broken or empty, and checks the first word against the list. If there's no list, anything still goes.
A search word that became a flag
The search tool put the pattern before the folder. A pattern that starts with a dash got read as one of the search program's own options. A double dash now goes in front, which tells the program that everything after it is just data.
A fence around the wrong yard
Every file action was limited to one fixed folder that fit one machine. Now the limit follows the project being worked on, with an optional list of extra folders.
Four checks, one per gap
Each gap has a test aimed at the old behavior:
- one allowed program gets through and an interpreter gets refused
- a failing command reports a failure and its exit code
- a search finds a word that starts with a dash
- a file inside the project is readable and a system file outside it isn't
Only the first word gets checked
The allowlist only looks at the first word. It decides which program starts. Whatever that program does once it's running is still up to the path limits and the timeouts.