Streaming

Set stream: true and the response becomes a server-sent event stream instead of one JSON body. Chat and legacy completions emit OpenAI's frames. A sequence of data: lines closed by data: [DONE]. POST /v1/messages emits Anthropic's named events instead, because an Anthropic client parses the event names rather than the payloads.

Consuming a stream

curl -N https://api.routehook.ai/v1/chat/completions \
  -H "Authorization: Bearer $ROUTEHOOK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Describe harbour fog." }],
    "stream": true,
    "stream_options": { "include_usage": true }
  }'

The frames

Every frame is one data: line and a blank line. Content chunks are OpenAI's chat.completion.chunk objects and carry the upstream's own completion id, which is stable for the whole response. The last content chunk carries a finish_reason and an empty delta. data: [DONE] always closes the stream, on success and on failure alike.

On the wire

data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-4o-mini","choices":[{"index":0,"delta":{"content":"Fog"},"finish_reason":null}]}

data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"req_7c41d9be","object":"chat.completion.chunk","created":1786312461,"model":"openai/gpt-4o-mini","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":42,"total_tokens":51,"cost":"0.0000181"}}

data: [DONE]

The model id is our slug

Every chunk is rewritten so that model is the slug you asked for, never the id an upstream knows the model by. A stream served by a fallback would otherwise report a model you never requested, halfway through a response you are already rendering, and your logging, your caching keys and your cost attribution all key off that field. Nothing on the response names the host that answered.

Usage and cost

Send stream_options: { include_usage: true } and one extra frame arrives before [DONE]: no choices, a full usage object, usage.cost as a number, and exact usage.cost_decimal. Without the opt-in nothing extra is sent. The alternative is GET /v1/generation?id=req_… afterwards.

When a stream dies halfway

A stream commits at its first byte. Once the headers are out the status is already 200 and there is no failing over to another target, so an upstream that dies mid-response cannot be retried behind your back. What arrives instead is an error frame, then [DONE], on the same 200 response. Both OpenAI SDKs raise on that frame rather than yielding it, so put the loop in a try. The bytes you already wrote are still yours.

A stream that failed after the first byte

data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-4o-mini","choices":[{"index":0,"delta":{"content":"Fog rolled"},"finish_reason":null}]}

data: {"error":{"code":"upstream_unavailable","message":"Connection reset by upstream after 1.8s.","request_id":"req_7c41d9be"}}

data: [DONE]

Before the first byte, it is an ordinary error

The failure path splits at the headers. A bad model, an empty routing chain, a missing key or an insufficient balance are all decided before anything is written, so they arrive as a normal JSON error with the right status (400, 401, 402, 409), and no stream is opened. Only failures after the first byte become error frames. A client that only handles error frames will miss every pre-flight refusal, and one that only handles status codes will treat a dead stream as a success.

Headers on a streamed response

Things that catch people out

• A proxy of your own can buffer a stream back into one lump. If output arrives all at once, the gateway is rarely the layer to blame. Check proxy_buffering first.
• Failover happens before the first byte only. The chain still runs for the pre-flight attempt; nothing can retry once a byte has shipped.
• [DONE] is not JSON. Parsing every frame's payload without checking for it is the most common integration bug on this route.
• Frames can be split across TCP reads. Buffer until a blank line before parsing, or use a client that does.
• A stream reserves credit exactly as a completion does, and an unaffordable stream is refused up front rather than cut off midway.

Legacy completions

POST /v1/completions streams in the same framing with a different payload: the object is text_completion and each chunk carries choices[].text rather than choices[].delta. Everything else on this page ([DONE], the error frame, the usage opt-in) is identical.

Anthropic messages

POST /v1/messages does not use this framing at all. It emits Anthropic's named events (message_start, content_block_start, content_block_delta, content_block_stop, message_delta and message_stop) each written as an event: line followed by its data: line, with no [DONE] sentinel. See the messages page for the full sequence.

Related