Flow
The handoff, step by step
1Ask the server to drop its model
Short timeout, errors ignored, so a load is never blocked.
2Unload the trained model for real
Clear references, run cleanup, empty the cache.
3Load onto the chosen card
Saved precision, direct placement, bad card setting fails fast.
4Out of memory anyway
Partial model thrown away, error says what to stop.
5Still slipping
Default model name only, training page chat skips it, full reload each switch.
A language model can run two ways in my coding assistant, and both of them act like the graphics card is theirs.
One is a model server that runs on its own and keeps a model loaded so replies are fast. The other lives inside my web app and loads a model I trained, so I can chat with it. They share one GPU. Until this change, neither knew the other was there, and the web app side had no way to unload a model at all.
I don't have a log of what made me fix this. The commit message is two words, and most of the change is me splitting one huge file into smaller ones. So this is from reading the diff.
Whoever is about to run kicks the other one off
The rule is simple. Before one side loads or answers, the other side lets go of the card. Loading my trained model first tells the server to drop its model, then waits a few seconds. Chatting with the trained model does the same. A normal chat unloads my trained model instead. One button clears both.
The request to the server gives up fast and ignores errors. I want a load to go ahead even if the server is down. The downside is that a failed request only shows up later, as an out-of-memory error.
An unload that frees nothing on the card
Setting a variable to nothing doesn't free GPU memory. The library holds on to its own cache, and any leftover reference keeps the model alive. So unload now does more than drop the model.
Unload drops the reference, runs a cleanup pass, and then empties the library's cache.
Every load runs that first.
Loading once, onto the right card
The card to use is a setting now, and a wrong one fails right away. Weights load straight onto the card at the precision the model was saved in. If the card runs out of memory anyway, the half-loaded model is thrown away and the error tells me to stop the other model first. A failed load never looks like a working one.
Chat replies also got cleaner. The model's own chat format is used when it has one, and only the new text is kept.
Where the handoff still leaks
The server is told to drop its default model, not whatever it has loaded. The training page has its own chat box that skips the handoff. The two tests check that the routes still answer after the file split, but none of them loads a real model.
And every time I flip between normal chat and my trained model, both sides reload from scratch. For quick back-and-forth testing, that wait adds up fast.