Flow
What happens to each line
1Partial line
Held back until the rest of it arrives.
2Blank or comment line
Skipped.
3Done marker
Skipped too. Reading continues until the server hangs up.
4Line that isn't valid JSON
Dropped, with no count and no warning.
5Line with text in it
Text added to the answer and sent to the screen.
Flow
Where the timer and the parser can still bite
1Overall timer
A slow but healthy answer can be cut off, and the error wipes what was on screen.
2Dropped bad lines
An answer could be missing words and still look complete.
3Done marker ignored
The app waits for the server to close the connection.
4No mid-stream failure test
Nothing proves the log stays clean when a stream breaks partway.
My desktop chat app showed nothing until the model was finished. You asked a question, and the window sat there until the whole reply came back at once from the model, which runs on another box in my rack. For long code answers, the app looked dead.
The app has two halves: a main process that does the network work and keeps a log of the conversation, and a page that only draws. With streaming, fragments of the answer now travel from one half to the other while the model is still writing.
The log feeds the next prompt
Each new question goes out with recent turns from that log, so the log isn't just a record; the model reads it back as context. Earlier that day I'd already had to strip old junk out of it before it went to the model.
A cut-off answer would be more junk. If a stream died halfway and the half got saved, the model would see a broken reply next time as if it had written it.
So the screen and the log stay apart. Pieces go to the screen as they come in. The log waits until the stream is over, and then saves the question and the full answer together:
const answer = soFar.trim();
if (!answer) throw new Error("empty reply");
await log.recordTurn(question, answer);
If anything fails before that point, the log never sees the exchange, neither the answer nor the question.
Reading the stream
The stream arrives as lines of text. A few rules handle them:
- Hold back a partial line until the rest of it arrives, and process what's left when the stream ends.
- Skip blank lines and comment lines, and skip the done marker too.
- Drop any line that isn't valid JSON.
- Pull the text from whichever common field has it, add it to the answer, and send it to the screen.
A bad status, a missing body, a timeout, or an empty answer all count as errors, so none of them gets saved.
The screen copy is a draft
The page shows a bubble with a blinking caret and updates it with each piece. When the call finishes, it swaps in the final message from the main process. If the call fails, the bubble turns into an error and shows the error text.
Both the old blocking call and the new streaming one build the request with the same shared code. That way they always ask the model the same question. If the streaming bridge is missing, the page falls back to the old call.
Checking that it really streams
Each result now reports how many pieces arrived. The smoke runs that start the real app and send a real prompt now fail if that number is zero. Otherwise a broken stream that fell back to the old call would pass, since an answer still shows up. The other new tests only check that certain strings exist in the code.
Where the timer and the parser can still bite
The big one: no test breaks a stream partway and checks that the log stayed clean.
The overall timer can cut off a long answer that was going fine, and then the error wipes out the part already on screen. Bad lines are dropped without a word, so an answer could be missing text and look complete. The done marker doesn't end the read, so the app waits for the server to hang up.