Choose a route for the workload
The class a request ran as is the
workload_class label on the server’s metrics.
Pick by who is waiting:
- A person is listening or talking now. Use
/v1/realtimeor a streaming speech route. Do not send live traffic asflex. - A program wants one answer soon. Use the synchronous routes without a tier.
- Nobody is waiting. Use
flexfor a handful of requests your code already sends one by one, and/v1/batchesfor anything large: a batch survives a restart, retries lines the server had no room for, and does not hold a connection open. A batch can call a webhook when it ends.
/v1/speech-to-text/jobs, /v1/text-to-speech) take no service_tier. Their durable jobs always run as batch work.
What the server does to protect live traffic
All of this is on by default.- Classes. Every request is admitted and scheduled under its class. Between models that share a device, device time is shared by class weight, and batch and background work have the smallest shares.
- Admission. Batch and background requests can never take the server’s last request slot (
IZWI_MAX_BATCH_REQUESTSandIZWI_MAX_BACKGROUND_REQUESTSdefault to the global limit less one). - The real-time reserve. While a model has live demand (a real-time or streaming request, or a real-time session operation, in the last 30 seconds), batch and background requests of that model may start only up to the model’s rows less a reserve. A live request then starts in a free row instead of waiting for batch work to finish. With no live demand batch work uses every row. A model that keeps more sequences than it decodes in one step (
IZWI_MAX_NUM_SEQSabove its rows) still leaves the reserve free among the extra sequences, which add no batch throughput, so the first live request after a quiet spell does not wait for a batch request to finish. - Class-aware encoding. When more transcription requests wait for a model’s audio encoder than one batch holds, the batch takes the requests of the higher classes first, so a live request does not wait behind a backlog of batch rows.
- Bounded batch steps. A long recording is transcribed in steps, and live requests are served between them. Parakeet ends its encode after each window group while its model has live demand.
- A separate real-time queue. Real-time requests and open sessions wait in a runtime queue of their own, so a flood of batch requests that fills the general queue does not refuse a call.
- Pacing. A speech stream that has produced more than
IZWI_STREAM_LEAD_SECSof audio beyond what has had time to play is scheduled from the batch class’s share until its lead falls to half of that. Nothing is throttled: on an idle device the stream still runs at full speed. - Bounded waiting. A chat or
/v1/audio/*request that finds the server at capacity waits up toIZWI_ADMISSION_QUEUE_TIMEOUT_SECSand is then refused with429and aRetry-Afterheader, before any stream starts. Aflexrequest’s refusal has coderesource_unavailableand a message startingResource Unavailable, so a client can retry later or fall back to the default tier. A/v1/batchesline refused this way is run again by the server rather than failed.
Settings
Set these in the server’s environment. The defaults suit a server that does both kinds of work.What to watch
ScrapeGET /v1/metrics/prometheus. The metrics reference describes every series.
Known limits
- VibeVoice-ASR and Nemotron 3.5 Transcribe read a whole recording in one pass. A batch request of either holds the device for that pass, and a live request of the same model waits for it. For strict live latency, run these two for recordings on a separate server or device.
- Kokoro streams are not paced. A streamed Kokoro request runs outside the scheduler that paces other speech streams. It still counts as its model’s live demand, so the reserve applies to batch Kokoro work beside it.
- First-party durable jobs take no tier. They always run as batch work.
- Running work is not preempted. A live request can wait for the batch job in front of it on the device to finish; bounded batch steps keep that wait short for the models that support them.
- One server holds a batch. Files and batches live in the server’s own store. Behind the gateway they are served only when its upstreams share that store, and each gateway tenant sees only its own once the upstreams run with
IZWI_TRUST_GATEWAY_TENANT_HEADER=true(Batch API).
Measure your capacity
No capacity figures are published yet: how many live sessions a device carries, alone and beside a batch load, has not been measured across models and hardware. Measure it on yours before you promise a latency.scripts/bench/run-mixed-load.py holds a fixed live load while it raises a batch load on the same server, and reports what each batch level does to live latency:
--slo-asr-first-partial-s, --slo-asr-final-s, --slo-tts-first-audio-s) and against the rise over the baseline (--slo-degradation). Raise --realtime-streams until the baseline no longer meets your targets to find how many live sessions the device carries, then read off the batch level it tolerates. scripts/bench/README.md has every option.