Flow
A check built on what the model reads
1One formatter
The data script and the server shape tool results with the same code.
2Hash the output
Run fixed sample results through it and hash the text plus the token list.
3Store it with the data
Save the value when the training set is built, and keep it with the model.
4Compare at start
The server works it out again and refuses to start if it differs.
Stopping my tool-calling model from starting if the code around it changes was the point of a check I built. Then I read the script that makes its training data, and it doesn't match that code at all.
The model is a small fine-tune that learns to call tools, like reading a file or running a command. When it runs, a wrapper watches for the token that starts a tool call, runs the tool, and feeds the result back in. That wrapper has rules. Each tool costs something, and a turn has a budget. Long files come back trimmed to the start and end, with a short header saying how big they were. Repeat the same request and it gets refused. A warning at the top of the rules file says any change means retraining.
That warning is right. The model reads those results while it learns, so the header and the trimming are part of what it learned. Change them later and it is reading text it never saw. Nothing errors. It just gets worse.
Freezing the runtime
So I hash a handful of things: the rules file, the token config, the code that formats results, and the sorted list of special tokens. Those roll up into one value that is meant to be saved at training time. A check is meant to compare it at startup and, if it differs, say which part changed and tell you to roll back or retrain.
The practice data looks different
The training data never goes through that wrapper. The data script uses a different way of writing a tool call. It lists tools the wrapper can't run. Its results are short strings typed by hand, with no header and no trimming. It counts the budget in calls, not cost. Even the error tokens are a different set. So the hash locks down a format the training data never shows, and it would still match.
It can't run yet
The token config is ignored by git, so a fresh copy of the repo can't build the hash. And nothing calls the check yet.
Check the text the model sees
If you build something like this, hash the text the model actually sees, and make the data script use the same formatter as the server.
Hash the rendered text of a fixed set of sample results together with the list of tool names.
Store that with the data and compare it when the server starts. Until the data script and the server share one formatter, I'm not sure what I'd be comparing it to.