Flow
One reply through the relay
Server steps first, then the browser cleanup every path has to reach.
1Headers go out first
Line-by-line JSON, no caching, no proxy buffering, buffers flushed. Success is already sent.
2Read whole lines only
Buffer the bytes, split on newlines, keep the unfinished tail for later.
3Send the whole reply so far
Lightly cleaned. Cut a copy at the marker, never the raw text.
4Finish with the strong filter
The last line is the real answer. Everything before it is a draft.
5Errors become a line
The status is spent, so write an error with a done flag and stop.
6Browser cleans up
Drop the typing note, unlock the input, remove an empty bubble, offer retry.
The chat page used to show nothing until the whole answer was done. Then the full reply dropped in at once. It worked, but a long answer felt dead.
So the chat endpoint switched to streaming. It asks the model for a stream and passes it to the browser one JSON line at a time. A second commit the same day fixed up the browser side. Speed was the dull part. The interesting part was everything the old request handled automatically that the stream had to rebuild by hand.
A 200 that means "I started"
The server sends its headers before it talks to the model. It turns off caching, tells the proxy not to hold anything back, and flushes its buffers. So the browser receives a success status before anything can fail, and that status can't be undone.
So errors go into the stream. If the model call fails, the server writes one last line with an error and a done flag. The browser's "bad status" check now only catches crashes before the first byte. If you count errors by status code, this endpoint will look fine while replies die halfway.
Every line carries the whole reply
Bytes from the model don't arrive in neat lines, so the server keeps a buffer and only parses complete lines. Each new piece of text is added to a running reply. The server cleans it and sends the whole thing so far. The browser just swaps the bubble's text each time.
When the model is done, the server runs the stronger content filter and sends that as the last line. What you see scroll by is a draft. The last line is the real answer.
The cleanup that cuts and then keeps going
One step cuts off a marker the model sometimes leaks, plus everything after it. The old code did that once, on the full reply. The new code cuts the running reply and saves the trimmed version. After that, the marker is gone, so new text just gets added on and goes through. I haven't seen it happen, but that's how I read the diff. The cleaner way keeps the raw text whole.
Each time, cut a copy of the raw text at the marker and send that, leaving the raw text itself untouched.
The browser has to close its own loop
The browser also reads whole lines only. The "typing" note is removed in a cleanup step, so it always goes away. The second commit allowed one request at a time, locked the input while it runs, and showed errors as an alert with a retry button. An empty bubble gets removed on failure. A partial one stays.
Where I think it still leaks
No tests came with this, so it's all from reading code. A stream that just stops looks finished. Bad lines are dropped with no record. Sending the whole reply each time gets heavy on long answers, though I haven't measured it.
And retry starts a fresh answer under the broken one, so one question can end up with two replies on screen, one of them cut short and neither of them labeled.