article
Two model runtimes shared my one GPU, so I made them take turns
The coding assistant I run has a resident model server and an in-process runtime for models I train, both on one graphics card. I added a handoff: evict the other side before loading, make unload actually free memory, load straight to the card, and turn out-of-memory into an instruction. Then the gaps I can still see in the diff.
Read more 4m read