One Izwi server can answer live traffic (phone calls, voice agents, streaming playback) and work through bulk jobs (recordings, long scripts) on the same device. The route a client calls tells the server which kind of work a request is, and the server keeps capacity free for the live kind. This page is for the operator who sizes and watches the server, and for the developer who chooses routes.

Choose a route for the workload

The class a request ran as is the workload_class label on the server’s metrics. Pick by who is waiting:
  • A person is listening or talking now. Use /v1/realtime or a streaming speech route. Do not send live traffic as flex.
  • A program wants one answer soon. Use the synchronous routes without a tier.
  • Nobody is waiting. Use flex for a handful of requests your code already sends one by one, and /v1/batches for anything large: a batch survives a restart, retries lines the server had no room for, and does not hold a connection open. A batch can call a webhook when it ends.
The first-party job routes (/v1/speech-to-text/jobs, /v1/text-to-speech) take no service_tier. Their durable jobs always run as batch work.

What the server does to protect live traffic

All of this is on by default.
  • Classes. Every request is admitted and scheduled under its class. Between models that share a device, device time is shared by class weight, and batch and background work have the smallest shares.
  • Admission. Batch and background requests can never take the server’s last request slot (IZWI_MAX_BATCH_REQUESTS and IZWI_MAX_BACKGROUND_REQUESTS default to the global limit less one).
  • The real-time reserve. While a model has live demand (a real-time or streaming request, or a real-time session operation, in the last 30 seconds), batch and background requests of that model may start only up to the model’s rows less a reserve. A live request then starts in a free row instead of waiting for batch work to finish. With no live demand batch work uses every row. A model that keeps more sequences than it decodes in one step (IZWI_MAX_NUM_SEQS above its rows) still leaves the reserve free among the extra sequences, which add no batch throughput, so the first live request after a quiet spell does not wait for a batch request to finish.
  • Class-aware encoding. When more transcription requests wait for a model’s audio encoder than one batch holds, the batch takes the requests of the higher classes first, so a live request does not wait behind a backlog of batch rows.
  • Bounded batch steps. A long recording is transcribed in steps, and live requests are served between them. Parakeet ends its encode after each window group while its model has live demand.
  • A separate real-time queue. Real-time requests and open sessions wait in a runtime queue of their own, so a flood of batch requests that fills the general queue does not refuse a call.
  • Pacing. A speech stream that has produced more than IZWI_STREAM_LEAD_SECS of audio beyond what has had time to play is scheduled from the batch class’s share until its lead falls to half of that. Nothing is throttled: on an idle device the stream still runs at full speed.
  • Bounded waiting. A chat or /v1/audio/* request that finds the server at capacity waits up to IZWI_ADMISSION_QUEUE_TIMEOUT_SECS and is then refused with 429 and a Retry-After header, before any stream starts. A flex request’s refusal has code resource_unavailable and a message starting Resource Unavailable, so a client can retry later or fall back to the default tier. A /v1/batches line refused this way is run again by the server rather than failed.
A running device job is never interrupted. Protection comes from keeping rows free and keeping batch jobs short, not from stopping work that has started.

Settings

Set these in the server’s environment. The defaults suit a server that does both kinds of work.

What to watch

Scrape GET /v1/metrics/prometheus. The metrics reference describes every series.

Known limits

  • VibeVoice-ASR and Nemotron 3.5 Transcribe read a whole recording in one pass. A batch request of either holds the device for that pass, and a live request of the same model waits for it. For strict live latency, run these two for recordings on a separate server or device.
  • Kokoro streams are not paced. A streamed Kokoro request runs outside the scheduler that paces other speech streams. It still counts as its model’s live demand, so the reserve applies to batch Kokoro work beside it.
  • First-party durable jobs take no tier. They always run as batch work.
  • Running work is not preempted. A live request can wait for the batch job in front of it on the device to finish; bounded batch steps keep that wait short for the models that support them.
  • One server holds a batch. Files and batches live in the server’s own store. Behind the gateway they are served only when its upstreams share that store, and each gateway tenant sees only its own once the upstreams run with IZWI_TRUST_GATEWAY_TENANT_HEADER=true (Batch API).

Measure your capacity

No capacity figures are published yet: how many live sessions a device carries, alone and beside a batch load, has not been measured across models and hardware. Measure it on yours before you promise a latency. scripts/bench/run-mixed-load.py holds a fixed live load while it raises a batch load on the same server, and reports what each batch level does to live latency:
Level 0 is the live load alone and is the baseline. Each step records live latency (first partial and final transcript, or first audio), batch throughput, refusals on both sides, and the change in the server’s metrics. The run checks each step against latency targets you can set (--slo-asr-first-partial-s, --slo-asr-final-s, --slo-tts-first-audio-s) and against the rise over the baseline (--slo-degradation). Raise --realtime-streams until the baseline no longer meets your targets to find how many live sessions the device carries, then read off the batch level it tolerates. scripts/bench/README.md has every option.

See also