- Which OS and hardware combinations are supported?
- Which shipped artifact types expose which backends?
- Which deployment targets are considered supported?
- Which API surfaces are stable vs preview?
Backend Matrix
Deployment Matrix
API Surface Maturity
The runtime exposes both compatibility APIs and first-party local workflow APIs under/v1.
When the server is running, open /docs for the local Scalar API reference or
/openapi.json for the raw OpenAPI document. The generated OpenAPI document
covers the stable OpenAI-compatible contract, /v1/responses preview routes,
readiness probes, and Scalar sidebar entries for preview first-party, operator,
and realtime route families. Detailed preview behavior is documented in the
API Reference.
CUDA Caveats
- Linux and Windows GitHub Releases keep public binary names unchanged:
izwiandizwi-serveron Linux,izwi.exeandizwi-server.exeon Windows. - Linux and Windows GitHub Release artifacts are CPU-only and must not contain CUDA runtime libraries or private CUDA binaries.
- Release installers do not replace the host NVIDIA driver. CUDA acceleration requires a compatible NVIDIA driver and CUDA-capable GPU.
- Source builds still require the CUDA toolkit and remain useful for development or fallback validation.
- The Docker CUDA image/profile is the CUDA distribution path for NVIDIA Linux hosts. It targets compute capability 8.0 (Ampere) and newer unless built with another
CUDA_COMPUTE_CAP. - On macOS 15+, the recommended GPU path is Metal, not CUDA. macOS 12-14 are CPU-only.
Chatterbox-500MandChatterbox-Multilingual-500Mbuild for CUDA but are compile-checked only: they have not yet run on an NVIDIA GPU. Their T3 speech-token models run in BF16 on CUDA devices with BF16, F16 on Metal and F32 on CPU, behind a load-time self-check that falls back to F32; their other networks run in F32.Chatterbox-500Mis verified end to end on CPU and Metal, andChatterbox-Multilingual-500Mon Metal.Maya1-3Bbuilds for CUDA but is compile-checked only: it has not yet run on an NVIDIA GPU. Its decoder runs in BF16 on CUDA devices with BF16 and F32 on CPU, each behind a load-time range probe that falls back to F32, and in F16 on Metal, where a failed probe refuses the load (F32 would need 13.2 GB of weights); SNAC runs in F32 everywhere.Maya1-3B-GGUFkeeps its decoder’s weights in Q8_0 and runs it in F32 on every backend, with no probe. Both are verified on Metal (16 GB M1 Pro): the decoder tracks upstream’s BF16 logits, the release server passes its end-to-end checks (voices, tags, streaming, cancellation, reload, Whisper transcripts), and requests at four and eight in flight sound exactly as they do alone; CPU runs the unit and layer-truncated parity tests only.Maya2-Native-2.5Bhas a CUDA path but has not yet run on an NVIDIA GPU. Its Qwen3 backbone runs in BF16 on CUDA devices with BF16, behind a load-time range probe that falls back to F32; in F16 on Metal, where a failed probe refuses the load (F32 would need 10 GB of weights); and in F32 on CPU. Its depth decoder runs in F16 on Metal and CUDA and in F32 on CPU or after an F32 fallback; Mimi runs in F32 everywhere, and the ECAPA speaker encoder for clones runs on the CPU in F32.LFM3-Audio-2B-GGUFandLFM3-Audio-2B-Q8_0-GGUFbuild for CUDA but are compile-checked only: they have not yet run on an NVIDIA GPU. On CUDA the quantized language model, speech encoder, MaskGIT depth decoder and LiquidCodec decoder use Candle CUDA kernels, while mel features, sampling and the inverse FFT run on the host. LFM3 Audio is verified end to end on real weights on CPU and Metal.Nemotron-3.5-Transcribe-30B-A4B-FP8builds for CUDA but is compile-checked only: it has not run on an NVIDIA GPU. It needs compute capability 8.0 or newer. See Nemotron 3.5 Transcribe.FishAudio-Drama-3needs compute capability 8.0 or newer on CUDA; an older GPU is refused before any weights are read. See Fish Audio Drama 3.
Nemotron 3.5 Transcribe
Nemotron-3.5-Transcribe-30B-A4B-FP8 runs on CPU, Metal and CUDA.
Each request holds about 48 MiB of decoder state plus its attention KV. On
CUDA a load reserves that state for 8 concurrent requests and a working copy
of each (about 0.75 GiB), beside 384 MiB of KV that every request shares;
more concurrent requests wait for a row. A load that the backend’s memory
budget cannot hold is refused before any weights are read.
IZWI_NEMOTRON_TRANSCRIBE_FP8_MODE=vllm-fp8 rounds activations to FP8
as vLLM’s W8A8 path does; it is a parity-testing mode, not a speed-up.
Fish Audio Drama 3
FishAudio-Drama-3 runs on CPU, Metal and CUDA. Its codec runs in F32 on every
backend.
Izwi estimates about 40 GB of memory for the model. Each request adds about
62.8 MiB of recurrent state plus its KV. A load that the backend’s memory
budget cannot hold is refused before any weights are read, with a message
naming what it needs.
Qwen3.8 CUDA weight residency
Qwen3.8-27B-FP8 keeps its official block-scaled FP8 checkpoint as the resident
format on CUDA by default. Projections stay as the checkpoint’s E4M3 bytes plus one
F32 scale per 128x128 block (about 24.7 GB), and the compact FP8 kernels apply
weight_scale_inv inside every matrix multiplication with 16-bit activations
(software FP8 decode, not FP8 Tensor Core arithmetic). Row-compatible projections
are packed into one matrix. Runtime diagnostics report
resident_representation: native_block_fp8_with_q8_fallback_dense_bf16 and
fp8_execution_mode: native_block_fp8_auto_self_checked.
The runtime enables this only after checks on the device, before load admission:
every projection shape must fit the kernel contract, the kernels must match a
reference at the checkpoint’s real inner dimensions, and a probe on real weights
read from the checkpoint must match a host reference built from the same bytes.
The load then checks each kind of projection, packs included, against the original
checkpoint bytes. If any check fails, the model loads with the compressed Q8_0
fallback instead (weight_scale_inv applied, then requantized to resident Candle
Q8_0, about 26.2 GB), reported as q8_0_requantized_projections_with_dense_bf16
and fp8_execution_mode: q8_0_fallback:<reason>. A check that fails during
loading fails that load, and the next load uses Q8_0 with matching memory
admission. Set IZWI_CUDA_PROJECTION_BACKEND=q8 or
IZWI_QWEN38_FP8_PROJECTIONS=legacy to choose Q8_0, and
IZWI_QWEN38_FP8_GEMV=legacy to keep the reference decode kernel.
With native block FP8, the LM head (2.54 GB in BF16) is also quantized at load to
the same block-FP8 format (1.27 GB), which the target logits and every MTP draft
read. This is a deliberate precision reduction: a host quality gate compares FP8
and BF16 logits on probe rows across the vocabulary before the head is used, and
fast_paths.fp8_lm_head reports the observed error. Logits stay BF16. Set
IZWI_QWEN38_FP8_LM_HEAD=legacy to keep the BF16 head.
Both representations are designed to fit 40/48 GB-class CUDA devices with a
resource-fitted context, subject to actual free VRAM and allocator headroom. Native
block FP8 is not CUDA-certified until the corresponding NVIDIA evidence is
retained.
Qwen3.8 Metal weight residency
On Apple Silicon,Qwen3.8-27B-FP8 also keeps its projections as the checkpoint’s
block-FP8 bytes (about 28.9 GiB resident including a 1 GiB prefill scratch, against
about 51 GiB for the F16 expansion), so 48 GB and 64 GB Macs can load it. Decode
rows (up to 8) run an FP8 matrix-vector kernel that decodes E4M3 in software with
F32 accumulation; longer prefill spans expand each projection to a transient F16
matrix for one matrix multiplication. The LM head and the other dense tensors stay
F16. Every kind of projection is checked against the original checkpoint bytes at
load; a projection whose matrix-vector kernel disagrees runs the expansion instead,
and a check that still fails fails that load, after which the next load expands to
F16. Diagnostics report resident_representation: native_block_fp8_with_dense_f16.
Set IZWI_METAL_PROJECTION_BACKEND=expanded (or [runtime.performance.metal] projection_backend = "expanded") to expand to F16 at load.
Qwen3.8 CPU weight residency
On the CPU,Qwen3.8-27B-FP8 keeps its block-FP8 projections and its LM head as
Q8_0 (the block scales applied, then requantized to 8-bit blocks of 32 values),
about 30.7 GiB resident instead of about 102 GiB for the F32 expansion; the
embedding and other dense tensors stay F32. Decode runs Candle’s Q8_0 matrix
multiplication, which also quantizes activations to 8 bits; prompt spans longer
than 8 tokens expand row tiles of the weights to F32 for one F32 matrix
multiplication. This changes CPU numerics: on the test fixtures, logits sit 2-4%
(relative L2) from the F32 expansion, and greedy tokens can flip at near-ties.
Diagnostics report resident_representation: q8_0_projections_with_dense_f32. Set
IZWI_CPU_PROJECTION_BACKEND=expanded (or [runtime.performance.cpu] projection_backend = "expanded") to keep the F32 expansion, and
IZWI_QWEN38_CPU_LM_HEAD=dense to keep only the head in F32.
Qwen3.8 multi-token prediction defaults
Native Qwen3.8 MTP is enabled by default with one draft token on CPU and Metal. On CUDA the default (cuda.mtp = "auto") leaves the MTP head out
whenever CUDA graph decode is planned (see below): until MTP verification
also runs in CUDA graphs, one graph launch per token is expected to beat
MTP’s eager draft and verify rounds. The head still loads on CUDA when graph
decode is not planned (IZWI_QWEN38_CUDA_GRAPHS=legacy,
IZWI_CUDA_DECODE_GRAPHS=off, IZWI_CUDA_FUSED_DECODE=off, or the Q8_0
projection fallback). Set IZWI_QWEN38_MTP=1 (or IZWI_CUDA_MTP=1) to load
the head on CUDA anyway, and IZWI_QWEN38_MTP=0 to disable MTP everywhere;
restart or reload the model after changing either. The runtime diagnostics
report the decision as optimization_evidence.mtp.decision. This CUDA
default awaits a fixed-prompt throughput comparison on the H100; until then,
IZWI_QWEN38_MTP=1 restores the previous behavior. The
IZWI_QWEN38_MTP_DRAFT_TOKENS setting controls proposal depth rather than
enablement; its default is 1, and supported explicit values are 1 through
3.
Qwen3.8 CUDA graph decode
On CUDA, single-row decode without an MTP head runs through captured CUDA graphs when the projections are native block FP8 and the fused decode kernels pass their load-time checks:- The whole step (the token embedding, all 64 layers, the final norm and the
block-FP8 LM head) is one graph, driven by a device step buffer that holds
the token, position, KV slot and block table. Where it does not apply (an
image position, or
IZWI_QWEN38_STEP_GRAPH=legacy), 17 graph segments run with attention eager between them. - Each graph is warmed eagerly, with its logits checked against the standard decode on a copy of the row state, then captured and verified twice against an eager run before it replays. A KV arena that grows forces a recapture. Any failure turns graph decode off for the process with the reason in the diagnostics, and decode continues on the eager fused path.
- A decode quantum of up to 8 tokens per engine step
(
IZWI_QWEN38_DECODE_QUANTUM, 1 to 64) keeps its width past a request’s soft scheduling deadline while the row decodes alone. Once the whole-step graph replays, the quantum queues step k+1 before it reads token k, choosing tokens with the device sampler (greedy or seeded sampling with penalties). Requests that ask for logprobs decode one step at a time. - Load admission reserves graph decode’s working memory, three recurrent-state slabs plus 64 MiB (about 513 MiB for the 27B checkpoint), whenever graph decode is planned.
cuda_graphs in the runtime diagnostics reports the resolution, the phase
(warm, capture, verify, replay or disabled), the counters, and the
whole-step graph and decode quantum. Switches: IZWI_QWEN38_CUDA_GRAPHS=legacy
(Qwen3.8 only), IZWI_CUDA_DECODE_GRAPHS=off (every model family),
IZWI_QWEN38_STEP_GRAPH=legacy (segments only),
IZWI_QWEN38_DECODE_PIPELINE=legacy (one step at a time) and
IZWI_QWEN38_FUSED_AB_GEMV=legacy (cuBLAS for the DeltaNet alpha/beta
projection). CUDA graph decode is not CUDA-certified until the corresponding
NVIDIA evidence is retained.
Managed Inference-State Support
This matrix describes ownership and provider classification in the current source tree. It is narrower than general model availability: a model can be available while a particular cache provider or hardware cell remains uncertified.
The runtime fails model loading when a required ABI-v2 operation set or exact
model/capability route is incomplete. Source/build eligibility never upgrades
a route to runtime-validated or performance-certified evidence. The runtime
does not silently switch back to a model-owned cache.
State topology by model family
The load path currently publishes ABI-v2 state as follows:
“ABI-v2 physical ownership” is not a claim of complete model-quality or
hardware performance certification. The exact model revision, capability,
backend, dtype, page geometry, attention semantics, and build feature cell must
pass its release lane.
CUDA context policy
CUDA uses the native context reported by the loaded chat checkpoint. CPU and Metal allocate the chat KV arena in full when the model loads, so without an explicitmax_sequence_length they fit the checkpoint’s context to the memory
budget and cap it at 32,768 tokens; an idle model therefore no longer holds its
whole advertised context in unified memory. Set max_sequence_length (or
IZWI_MAX_SEQUENCE_LENGTH) to choose a longer context that the memory budget
can hold. Chat requests claim KV as they generate rather than reserving their
whole output budget, so concurrent requests share one arena. The current
native CUDA ceilings are:
Optional YaRN extensions are not included: the local adapters do not implement
their scaling parameters. Qwen3.8 fits its effective context to remaining
resources without changing the 262,144-token logical maximum. Other CUDA chat
routes reserve their selected paged KV capacity before publishing a model as
ready. Explicit context requests still fail when the requested state cannot
fit; the runtime does not overcommit VRAM.
For audio generation, automatic output budgets remain bounded. Explicit CUDA
requests can use the 32,768-token LFM2.5 Audio and Fish S2 contexts; Voxtral TTS
accepts the upstream deployment ceiling of 2,048 output frames. CPU and Metal
retain their existing limits. On CUDA, VibeVoice ASR runs input of up to
5 minutes in a single pass and splits longer input into chunks, because a
10-minute single pass ran out of memory on an 80 GB H100.
IZWI_VIBEVOICE_ASR_CUDA_MAX_AUDIO_SECS sets that single-pass limit, up to
the model’s documented 60-minute window. See the
LFM2.5 Audio model card,
Fish S2 config,
Voxtral deployment,
and VibeVoice ASR model card.
Qwen3 ASR accepts up to 1,200 seconds per CUDA transcription chunk; CPU and
Metal retain the checkpoint preprocessor window, and the CUDA route does not
silently truncate audio at that limit. The Qwen3 forced aligner takes one
window of up to 180 seconds on every backend, so its timings do not depend on
the backend; longer audio is aligned in sub-windows cut at quiet points.
Voxtral Realtime CUDA offline decoding uses the loaded model position range
(131,072 tokens in the current checkpoint) instead of the portable 1,024-frame
service limit. Its physical paged cache rotates the 8,192-token attention
window with one spare page; inputs beyond the loaded position range fail
explicitly instead of silently dropping the tail.
Architectural processing windows are not expanded. Whisper still processes
30-second encoder windows, Kokoro still uses at most 510 phonemes per model
chunk, and streaming ASR/diarization families retain their bounded working
buffers while the runtime orchestrates longer inputs.
Cache policy support
Eligible CUDA routes automatically use resident block-table metadata,
admission-grown physical backing, device/shape-keyed attention selection,
bounded sampling readback, VRAM-tiered continuous batches, and stable one-pass
decode graph buckets. Graph keys include exact K/V and metadata tensor
identities, geometry, dtype, scale, and softcap. Partitioned decode, FP8
storage, unobserved/pre-Ampere devices, and any capture/replay failure retain
eager native decode. CPU and Metal do not enter this policy.
For configuration, counters, benchmarks, and rollback, see
KV Cache Operations.
Verification Guidance
Use the following expectations when validating a host:- Apple Silicon, macOS 15+: build or install a Metal-capable binary and run with
--backend metalorIZWI_BACKEND=metal. - macOS 12-14: run with
--backend cpu;autoand explicitmetalrequests fall back to CPU. - Linux/Windows GitHub Release: run
izwi serve --backend cpu, thenizwi status --detailed. - Docker CUDA on NVIDIA Linux hosts: run
docker compose --profile cuda up -d izwi-cuda, then confirm the container selects CUDA through/v1/healthorizwi status --detailed. With the production overlay, sendAuthorization: Bearer <key>(or export the sameIZWI_API_KEYfor the CLI). Eligible Candle FlashAttention and Qwen3/Qwen3.5 RoPE CUDA routes activate automatically; their environment variables remain explicit rollback switches. Qwen3.8 is an independently gated family and must report its own execution path and evidence. - Linux/Windows source build for CUDA: build with
cargo build --release -p izwi-cli -p izwi-server --features cuda, then run with--backend cudaorIZWI_BACKEND=cuda. Thecudawrapper includes Candle FlashAttention, whilecudnnadditionally enables matching Candle/cuDNN convolution paths.