Rate Limits

Requests are metered per minute: 600 a minute unless your account is set otherwise. Every /v1 response to a call carrying your key reports where the meter stands, not only the ones that fail, so a client can slow down before it is refused instead of after. Going over answers 429 rate_limited with a Retry-After header.

The headers

Which ceiling you hit

rate_limited covers two different limits, and the fix is not the same for both. X-Routehook-Limit-Type: requests means too many calls in a minute: the meter refills on its own, so waiting out Retry-After is the whole remedy. X-Routehook-Limit-Type: concurrency means too much long-running work is in flight at once, ten jobs by default. No amount of waiting refills that one; a job has to finish first, and a retry loop against it will keep failing until one does. Wait for a job to reach a terminal state, then submit the next.

Whose limit it is

The meter belongs to the account, not to the key. Every key on an account draws on the same ceiling, so minting a second key buys no extra throughput. It buys separate revocation, which is a different and worthwhile thing. A project may carry its own limit, and where it does that limit replaces the account default for calls made under it. GET /v1/key publishes the ceiling in force for the key you are holding, as rate_limit: { requests, interval }, so a client can read its own budget rather than assume one.

Backing off properly

• Honour Retry-After when it is there. It is the server's own estimate of when the window reopens, not a floor to improve on.
• Otherwise back off exponentially, with full jitter. A fixed delay synchronises every client that failed in the same second, and they all come back in the same second.
• Cap the delay and the attempt count. Nothing is gained by a fourteenth retry of a request nobody is waiting for any more.
• Do not retry 400, 401, 402 or 409. The same request fails the same way; only 429 and 503 are worth sending again.
• Treat concurrency as a queue, not a clock. Wait for work to finish rather than for time to pass.

Retry with jitter

const RETRYABLE = new Set([429, 503]);

async function callWithRetry(body, attempts = 5) {
  for (let attempt = 0; attempt < attempts; attempt++) {
    const res = await fetch("https://api.routehook.ai/v1/chat/completions", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.ROUTEHOOK_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });

    if (res.ok) return res.json();

    const { error } = await res.json();
    if (!RETRYABLE.has(res.status) || attempt === attempts - 1) {
      throw new Error(`${error.code}: ${error.message} (${error.request_id})`);
    }

    // The server's estimate first; full jitter under a growing cap otherwise.
    const retryAfter = Number(res.headers.get("Retry-After")) || 0;
    const ceiling = Math.min(2 ** attempt, 30) * 1000;
    const wait = retryAfter > 0 ? retryAfter * 1000 : Math.random() * ceiling;

    await new Promise((resolve) => setTimeout(resolve, wait));
  }
}

Streaming and long jobs

A streamed completion is one request against the meter however long it runs and however much it costs. Rate limits count calls, not tokens. Work that takes minutes belongs on the queue, where the ceiling is on how much runs at once rather than on how often you ask, and where a 429 tells you to wait for a job rather than for a window.

Asking for more

Ceilings are set per account and can be raised. Read the current one from GET /v1/key first, then ask through support with the request ids of the calls that were refused. The ids make the pattern legible in a way a description of it cannot.

Related