# Streaming

> Server-sent events for chat, legacy completions and Anthropic messages.

Section: Core API. Canonical page: https://routehook.ai/docs/streaming

Set `stream: true` and the response becomes a server-sent event stream instead of one JSON body. Chat and legacy completions emit OpenAI's frames. A sequence of `data:` lines closed by `data: [DONE]`. `POST /v1/messages` emits Anthropic's named events instead, because an Anthropic client parses the event names rather than the payloads.

## Consuming a stream

**cURL**

```bash
curl -N https://api.routehook.ai/v1/chat/completions \
  -H "Authorization: Bearer $ROUTEHOOK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [{ "role": "user", "content": "Describe harbour fog." }],
    "stream": true,
    "stream_options": { "include_usage": true }
  }'
```

**JavaScript**

```javascript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.routehook.ai/v1",
  apiKey: process.env.ROUTEHOOK_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Describe harbour fog." }],
  stream: true,
  stream_options: { include_usage: true },
});

try {
  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta?.content;
    if (delta) process.stdout.write(delta);

    // The final frame carries no choices at all. Only usage, and cost.
    if (chunk.usage) console.log("\ncost", chunk.usage.cost, "USD");
  }
} catch (error) {
  // An error frame is raised here, after real bytes have already arrived.
  // Keep what you have: it was generated, and it has been billed.
  console.error("\nstream failed:", error.message);
}
```

**Python**

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.routehook.ai/v1",
    api_key=os.environ["ROUTEHOOK_API_KEY"],
)

stream = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[{"role": "user", "content": "Describe harbour fog."}],
    stream=True,
    stream_options={"include_usage": True},
)

try:
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)

        # The final frame carries no choices at all. Only usage, and cost.
        if chunk.usage:
            print("\ncost", chunk.usage.cost, "USD")
except Exception as error:
    # An error frame is raised here, after real bytes have already arrived.
    print("\nstream failed:", error)
```

> **Note: cost is additive, so typed clients do not declare it**
>
> `usage.cost` is our field, not OpenAI's, so an OpenAI SDK's own types have no property for it. It is on the object at runtime; in TypeScript, read it through a cast rather than expecting the SDK to type it. The Python client keeps unknown fields as attributes, so `chunk.usage.cost` works as written.

## The frames

Every frame is one `data:` line and a blank line. Content chunks are OpenAI's `chat.completion.chunk` objects and carry the upstream's own completion id, which is stable for the whole response. The last content chunk carries a `finish_reason` and an empty delta. `data: [DONE]` always closes the stream, on success and on failure alike.

## On the wire

**text/event-stream**

```text
data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-5.4-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-5.4-mini","choices":[{"index":0,"delta":{"content":"Fog"},"finish_reason":null}]}

data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-5.4-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"req_7c41d9be","object":"chat.completion.chunk","created":1786312461,"model":"openai/gpt-5.4-mini","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":42,"total_tokens":51,"cost":"0.0000181"}}

data: [DONE]
```

## The model id is our slug

Every chunk is rewritten so that `model` is the slug you asked for, never the id an upstream knows the model by. A stream served by a fallback would otherwise report a model you never requested, halfway through a response you are already rendering, and your logging, your caching keys and your cost attribution all key off that field. Nothing on the response names the host that answered.

## Usage and cost

Send `stream_options: { include_usage: true }` and one extra frame arrives before `[DONE]`: no choices, a full `usage` object, `usage.cost` as a number, and exact `usage.cost_decimal`. Without the opt-in nothing extra is sent. The alternative is `GET /v1/generation?id=req_…` afterwards.

> **Note: The usage frame carries a different id**
>
> Content chunks carry the upstream's completion id. The usage frame carries **our** request id, the `req_…` value also returned in `X-Routehook-Request-Id`: because that is the id `GET /v1/generation` and every support lookup are made on. Do not assume the two match.

## When a stream dies halfway

A stream commits at its first byte. Once the headers are out the status is already 200 and there is no failing over to another target, so an upstream that dies mid-response cannot be retried behind your back. What arrives instead is an error frame, then `[DONE]`, on the same 200 response. Both OpenAI SDKs raise on that frame rather than yielding it, so put the loop in a try. The bytes you already wrote are still yours.

## A stream that failed after the first byte

**text/event-stream**

```text
data: {"id":"chatcmpl_5f81c0","object":"chat.completion.chunk","created":1786312455,"model":"openai/gpt-5.4-mini","choices":[{"index":0,"delta":{"content":"Fog rolled"},"finish_reason":null}]}

data: {"error":{"code":"upstream_unavailable","message":"Connection reset by upstream after 1.8s.","request_id":"req_7c41d9be"}}

data: [DONE]
```

> **Warning: A half-delivered stream is still billed**
>
> You are holding real bytes: the tokens in them were generated, and we were charged for them upstream. So a stream that fails halfway settles for what was metered up to the failure, and is recorded as a failed request. No usage frame is emitted on that path (the error frame is the last thing before `[DONE]`), so read the charge from `GET /v1/generation?id=req_…` using the request id in the error. The same applies when **you** hang up: closing the connection aborts the generation upstream so nothing further is produced, and what was already metered is charged.

## Before the first byte, it is an ordinary error

The failure path splits at the headers. A bad model, an empty routing chain, a missing key or an insufficient balance are all decided before anything is written, so they arrive as a normal JSON error with the right status (`400`, `401`, `402`, `409`), and no stream is opened. Only failures after the first byte become error frames. A client that only handles error frames will miss every pre-flight refusal, and one that only handles status codes will treat a dead stream as a success.

## Headers on a streamed response

| HEADER | VALUE |
| --- | --- |
| Content-Type | text/event-stream; charset=utf-8 |
| Cache-Control | no-cache, no-transform |
| X-Accel-Buffering | no. Stops an nginx in front of us collecting the stream and delivering it in one lump |
| X-Routehook-Request-Id | The request id, before the first frame |
| X-Routehook-Cost | Not sent on a stream. The cost is unknown when the headers go out. |

## Things that catch people out

- A proxy of your own can buffer a stream back into one lump. If output arrives all at once, the gateway is rarely the layer to blame. Check `proxy_buffering` first.
- Failover happens before the first byte only. The chain still runs for the pre-flight attempt; nothing can retry once a byte has shipped.
- `[DONE]` is not JSON. Parsing every frame's payload without checking for it is the most common integration bug on this route.
- Frames can be split across TCP reads. Buffer until a blank line before parsing, or use a client that does.
- A stream reserves credit exactly as a completion does, and an unaffordable stream is refused up front rather than cut off midway.

## Legacy completions

`POST /v1/completions` streams in the same framing with a different payload: the object is `text_completion` and each chunk carries `choices[].text` rather than `choices[].delta`. Everything else on this page (`[DONE]`, the error frame, the usage opt-in) is identical.

## Anthropic messages

`POST /v1/messages` does not use this framing at all. It emits Anthropic's named events (`message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta` and `message_stop`) each written as an `event:` line followed by its `data:` line, with no `[DONE]` sentinel. See the messages page for the full sequence.

## Related

- [Chat](/docs/chat): The request this page streams, and its parameters.
- [Messages](/docs/messages): Anthropic's event framing, emitted by /v1/messages.
- [Errors](/docs/errors): The eight codes, and what each one means you should do.