This page is the public support contract for Izwi’s current runtime surfaces. It answers four questions:
  1. Which OS and hardware combinations are supported?
  2. Which shipped artifact types expose which backends?
  3. Which deployment targets are considered supported?
  4. Which API surfaces are stable vs preview?
If another page says something different, this page should win.

Backend Matrix


Deployment Matrix


API Surface Maturity

The runtime exposes both compatibility APIs and first-party local workflow APIs under /v1. When the server is running, open /docs for the local Scalar API reference or /openapi.json for the raw OpenAPI document. The generated OpenAPI document covers the stable OpenAI-compatible contract, /v1/responses preview routes, readiness probes, and Scalar sidebar entries for preview first-party, operator, and realtime route families. Detailed preview behavior is documented in the API Reference.

CUDA Caveats

  • Linux and Windows GitHub Releases keep public binary names unchanged: izwi and izwi-server on Linux, izwi.exe and izwi-server.exe on Windows.
  • Linux and Windows GitHub Release artifacts are CPU-only and must not contain CUDA runtime libraries or private CUDA binaries.
  • Release installers do not replace the host NVIDIA driver. CUDA acceleration requires a compatible NVIDIA driver and CUDA-capable GPU.
  • Source builds still require the CUDA toolkit and remain useful for development or fallback validation.
  • The Docker CUDA image/profile is the CUDA distribution path for NVIDIA Linux hosts. It targets compute capability 8.0 (Ampere) and newer unless built with another CUDA_COMPUTE_CAP.
  • On macOS 15+, the recommended GPU path is Metal, not CUDA. macOS 12-14 are CPU-only.
  • Chatterbox-500M and Chatterbox-Multilingual-500M build for CUDA but are compile-checked only: they have not yet run on an NVIDIA GPU. Their T3 speech-token models run in BF16 on CUDA devices with BF16, F16 on Metal and F32 on CPU, behind a load-time self-check that falls back to F32; their other networks run in F32. Chatterbox-500M is verified end to end on CPU and Metal, and Chatterbox-Multilingual-500M on Metal.
  • Maya1-3B builds for CUDA but is compile-checked only: it has not yet run on an NVIDIA GPU. Its decoder runs in BF16 on CUDA devices with BF16 and F32 on CPU, each behind a load-time range probe that falls back to F32, and in F16 on Metal, where a failed probe refuses the load (F32 would need 13.2 GB of weights); SNAC runs in F32 everywhere. Maya1-3B-GGUF keeps its decoder’s weights in Q8_0 and runs it in F32 on every backend, with no probe. Both are verified on Metal (16 GB M1 Pro): the decoder tracks upstream’s BF16 logits, the release server passes its end-to-end checks (voices, tags, streaming, cancellation, reload, Whisper transcripts), and requests at four and eight in flight sound exactly as they do alone; CPU runs the unit and layer-truncated parity tests only.
  • Maya2-Native-2.5B has a CUDA path but has not yet run on an NVIDIA GPU. Its Qwen3 backbone runs in BF16 on CUDA devices with BF16, behind a load-time range probe that falls back to F32; in F16 on Metal, where a failed probe refuses the load (F32 would need 10 GB of weights); and in F32 on CPU. Its depth decoder runs in F16 on Metal and CUDA and in F32 on CPU or after an F32 fallback; Mimi runs in F32 everywhere, and the ECAPA speaker encoder for clones runs on the CPU in F32.
  • LFM3-Audio-2B-GGUF and LFM3-Audio-2B-Q8_0-GGUF build for CUDA but are compile-checked only: they have not yet run on an NVIDIA GPU. On CUDA the quantized language model, speech encoder, MaskGIT depth decoder and LiquidCodec decoder use Candle CUDA kernels, while mel features, sampling and the inverse FFT run on the host. LFM3 Audio is verified end to end on real weights on CPU and Metal.
  • Nemotron-3.5-Transcribe-30B-A4B-FP8 builds for CUDA but is compile-checked only: it has not run on an NVIDIA GPU. It needs compute capability 8.0 or newer. See Nemotron 3.5 Transcribe.
  • FishAudio-Drama-3 needs compute capability 8.0 or newer on CUDA; an older GPU is refused before any weights are read. See Fish Audio Drama 3.

Nemotron 3.5 Transcribe

Nemotron-3.5-Transcribe-30B-A4B-FP8 runs on CPU, Metal and CUDA. Each request holds about 48 MiB of decoder state plus its attention KV. On CUDA a load reserves that state for 8 concurrent requests and a working copy of each (about 0.75 GiB), beside 384 MiB of KV that every request shares; more concurrent requests wait for a row. A load that the backend’s memory budget cannot hold is refused before any weights are read. IZWI_NEMOTRON_TRANSCRIBE_FP8_MODE=vllm-fp8 rounds activations to FP8 as vLLM’s W8A8 path does; it is a parity-testing mode, not a speed-up.

Fish Audio Drama 3

FishAudio-Drama-3 runs on CPU, Metal and CUDA. Its codec runs in F32 on every backend. Izwi estimates about 40 GB of memory for the model. Each request adds about 62.8 MiB of recurrent state plus its KV. A load that the backend’s memory budget cannot hold is refused before any weights are read, with a message naming what it needs.

Qwen3.8 CUDA weight residency

Qwen3.8-27B-FP8 keeps its official block-scaled FP8 checkpoint as the resident format on CUDA by default. Projections stay as the checkpoint’s E4M3 bytes plus one F32 scale per 128x128 block (about 24.7 GB), and the compact FP8 kernels apply weight_scale_inv inside every matrix multiplication with 16-bit activations (software FP8 decode, not FP8 Tensor Core arithmetic). Row-compatible projections are packed into one matrix. Runtime diagnostics report resident_representation: native_block_fp8_with_q8_fallback_dense_bf16 and fp8_execution_mode: native_block_fp8_auto_self_checked. The runtime enables this only after checks on the device, before load admission: every projection shape must fit the kernel contract, the kernels must match a reference at the checkpoint’s real inner dimensions, and a probe on real weights read from the checkpoint must match a host reference built from the same bytes. The load then checks each kind of projection, packs included, against the original checkpoint bytes. If any check fails, the model loads with the compressed Q8_0 fallback instead (weight_scale_inv applied, then requantized to resident Candle Q8_0, about 26.2 GB), reported as q8_0_requantized_projections_with_dense_bf16 and fp8_execution_mode: q8_0_fallback:<reason>. A check that fails during loading fails that load, and the next load uses Q8_0 with matching memory admission. Set IZWI_CUDA_PROJECTION_BACKEND=q8 or IZWI_QWEN38_FP8_PROJECTIONS=legacy to choose Q8_0, and IZWI_QWEN38_FP8_GEMV=legacy to keep the reference decode kernel. With native block FP8, the LM head (2.54 GB in BF16) is also quantized at load to the same block-FP8 format (1.27 GB), which the target logits and every MTP draft read. This is a deliberate precision reduction: a host quality gate compares FP8 and BF16 logits on probe rows across the vocabulary before the head is used, and fast_paths.fp8_lm_head reports the observed error. Logits stay BF16. Set IZWI_QWEN38_FP8_LM_HEAD=legacy to keep the BF16 head. Both representations are designed to fit 40/48 GB-class CUDA devices with a resource-fitted context, subject to actual free VRAM and allocator headroom. Native block FP8 is not CUDA-certified until the corresponding NVIDIA evidence is retained.

Qwen3.8 Metal weight residency

On Apple Silicon, Qwen3.8-27B-FP8 also keeps its projections as the checkpoint’s block-FP8 bytes (about 28.9 GiB resident including a 1 GiB prefill scratch, against about 51 GiB for the F16 expansion), so 48 GB and 64 GB Macs can load it. Decode rows (up to 8) run an FP8 matrix-vector kernel that decodes E4M3 in software with F32 accumulation; longer prefill spans expand each projection to a transient F16 matrix for one matrix multiplication. The LM head and the other dense tensors stay F16. Every kind of projection is checked against the original checkpoint bytes at load; a projection whose matrix-vector kernel disagrees runs the expansion instead, and a check that still fails fails that load, after which the next load expands to F16. Diagnostics report resident_representation: native_block_fp8_with_dense_f16. Set IZWI_METAL_PROJECTION_BACKEND=expanded (or [runtime.performance.metal] projection_backend = "expanded") to expand to F16 at load.

Qwen3.8 CPU weight residency

On the CPU, Qwen3.8-27B-FP8 keeps its block-FP8 projections and its LM head as Q8_0 (the block scales applied, then requantized to 8-bit blocks of 32 values), about 30.7 GiB resident instead of about 102 GiB for the F32 expansion; the embedding and other dense tensors stay F32. Decode runs Candle’s Q8_0 matrix multiplication, which also quantizes activations to 8 bits; prompt spans longer than 8 tokens expand row tiles of the weights to F32 for one F32 matrix multiplication. This changes CPU numerics: on the test fixtures, logits sit 2-4% (relative L2) from the F32 expansion, and greedy tokens can flip at near-ties. Diagnostics report resident_representation: q8_0_projections_with_dense_f32. Set IZWI_CPU_PROJECTION_BACKEND=expanded (or [runtime.performance.cpu] projection_backend = "expanded") to keep the F32 expansion, and IZWI_QWEN38_CPU_LM_HEAD=dense to keep only the head in F32.

Qwen3.8 multi-token prediction defaults

Native Qwen3.8 MTP is enabled by default with one draft token on CPU and Metal. On CUDA the default (cuda.mtp = "auto") leaves the MTP head out whenever CUDA graph decode is planned (see below): until MTP verification also runs in CUDA graphs, one graph launch per token is expected to beat MTP’s eager draft and verify rounds. The head still loads on CUDA when graph decode is not planned (IZWI_QWEN38_CUDA_GRAPHS=legacy, IZWI_CUDA_DECODE_GRAPHS=off, IZWI_CUDA_FUSED_DECODE=off, or the Q8_0 projection fallback). Set IZWI_QWEN38_MTP=1 (or IZWI_CUDA_MTP=1) to load the head on CUDA anyway, and IZWI_QWEN38_MTP=0 to disable MTP everywhere; restart or reload the model after changing either. The runtime diagnostics report the decision as optimization_evidence.mtp.decision. This CUDA default awaits a fixed-prompt throughput comparison on the H100; until then, IZWI_QWEN38_MTP=1 restores the previous behavior. The IZWI_QWEN38_MTP_DRAFT_TOKENS setting controls proposal depth rather than enablement; its default is 1, and supported explicit values are 1 through 3.

Qwen3.8 CUDA graph decode

On CUDA, single-row decode without an MTP head runs through captured CUDA graphs when the projections are native block FP8 and the fused decode kernels pass their load-time checks:
  • The whole step (the token embedding, all 64 layers, the final norm and the block-FP8 LM head) is one graph, driven by a device step buffer that holds the token, position, KV slot and block table. Where it does not apply (an image position, or IZWI_QWEN38_STEP_GRAPH=legacy), 17 graph segments run with attention eager between them.
  • Each graph is warmed eagerly, with its logits checked against the standard decode on a copy of the row state, then captured and verified twice against an eager run before it replays. A KV arena that grows forces a recapture. Any failure turns graph decode off for the process with the reason in the diagnostics, and decode continues on the eager fused path.
  • A decode quantum of up to 8 tokens per engine step (IZWI_QWEN38_DECODE_QUANTUM, 1 to 64) keeps its width past a request’s soft scheduling deadline while the row decodes alone. Once the whole-step graph replays, the quantum queues step k+1 before it reads token k, choosing tokens with the device sampler (greedy or seeded sampling with penalties). Requests that ask for logprobs decode one step at a time.
  • Load admission reserves graph decode’s working memory, three recurrent-state slabs plus 64 MiB (about 513 MiB for the 27B checkpoint), whenever graph decode is planned.
cuda_graphs in the runtime diagnostics reports the resolution, the phase (warm, capture, verify, replay or disabled), the counters, and the whole-step graph and decode quantum. Switches: IZWI_QWEN38_CUDA_GRAPHS=legacy (Qwen3.8 only), IZWI_CUDA_DECODE_GRAPHS=off (every model family), IZWI_QWEN38_STEP_GRAPH=legacy (segments only), IZWI_QWEN38_DECODE_PIPELINE=legacy (one step at a time) and IZWI_QWEN38_FUSED_AB_GEMV=legacy (cuBLAS for the DeltaNet alpha/beta projection). CUDA graph decode is not CUDA-certified until the corresponding NVIDIA evidence is retained.

Managed Inference-State Support

This matrix describes ownership and provider classification in the current source tree. It is narrower than general model availability: a model can be available while a particular cache provider or hardware cell remains uncertified. The runtime fails model loading when a required ABI-v2 operation set or exact model/capability route is incomplete. Source/build eligibility never upgrades a route to runtime-validated or performance-certified evidence. The runtime does not silently switch back to a model-owned cache.

State topology by model family

The load path currently publishes ABI-v2 state as follows: “ABI-v2 physical ownership” is not a claim of complete model-quality or hardware performance certification. The exact model revision, capability, backend, dtype, page geometry, attention semantics, and build feature cell must pass its release lane.

CUDA context policy

CUDA uses the native context reported by the loaded chat checkpoint. CPU and Metal allocate the chat KV arena in full when the model loads, so without an explicit max_sequence_length they fit the checkpoint’s context to the memory budget and cap it at 32,768 tokens; an idle model therefore no longer holds its whole advertised context in unified memory. Set max_sequence_length (or IZWI_MAX_SEQUENCE_LENGTH) to choose a longer context that the memory budget can hold. Chat requests claim KV as they generate rather than reserving their whole output budget, so concurrent requests share one arena. The current native CUDA ceilings are: Optional YaRN extensions are not included: the local adapters do not implement their scaling parameters. Qwen3.8 fits its effective context to remaining resources without changing the 262,144-token logical maximum. Other CUDA chat routes reserve their selected paged KV capacity before publishing a model as ready. Explicit context requests still fail when the requested state cannot fit; the runtime does not overcommit VRAM. For audio generation, automatic output budgets remain bounded. Explicit CUDA requests can use the 32,768-token LFM2.5 Audio and Fish S2 contexts; Voxtral TTS accepts the upstream deployment ceiling of 2,048 output frames. CPU and Metal retain their existing limits. On CUDA, VibeVoice ASR runs input of up to 5 minutes in a single pass and splits longer input into chunks, because a 10-minute single pass ran out of memory on an 80 GB H100. IZWI_VIBEVOICE_ASR_CUDA_MAX_AUDIO_SECS sets that single-pass limit, up to the model’s documented 60-minute window. See the LFM2.5 Audio model card, Fish S2 config, Voxtral deployment, and VibeVoice ASR model card. Qwen3 ASR accepts up to 1,200 seconds per CUDA transcription chunk; CPU and Metal retain the checkpoint preprocessor window, and the CUDA route does not silently truncate audio at that limit. The Qwen3 forced aligner takes one window of up to 180 seconds on every backend, so its timings do not depend on the backend; longer audio is aligned in sub-windows cut at quiet points. Voxtral Realtime CUDA offline decoding uses the loaded model position range (131,072 tokens in the current checkpoint) instead of the portable 1,024-frame service limit. Its physical paged cache rotates the 8,192-token attention window with one spare page; inputs beyond the loaded position range fail explicitly instead of silently dropping the tail. Architectural processing windows are not expanded. Whisper still processes 30-second encoder windows, Kokoro still uses at most 510 phonemes per model chunk, and streaming ASR/diarization families retain their bounded working buffers while the runtime orchestrates longer inputs.

Cache policy support

Eligible CUDA routes automatically use resident block-table metadata, admission-grown physical backing, device/shape-keyed attention selection, bounded sampling readback, VRAM-tiered continuous batches, and stable one-pass decode graph buckets. Graph keys include exact K/V and metadata tensor identities, geometry, dtype, scale, and softcap. Partitioned decode, FP8 storage, unobserved/pre-Ampere devices, and any capture/replay failure retain eager native decode. CPU and Metal do not enter this policy. For configuration, counters, benchmarks, and rollback, see KV Cache Operations.

Verification Guidance

Use the following expectations when validating a host:
  • Apple Silicon, macOS 15+: build or install a Metal-capable binary and run with --backend metal or IZWI_BACKEND=metal.
  • macOS 12-14: run with --backend cpu; auto and explicit metal requests fall back to CPU.
  • Linux/Windows GitHub Release: run izwi serve --backend cpu, then izwi status --detailed.
  • Docker CUDA on NVIDIA Linux hosts: run docker compose --profile cuda up -d izwi-cuda, then confirm the container selects CUDA through /v1/health or izwi status --detailed. With the production overlay, send Authorization: Bearer <key> (or export the same IZWI_API_KEY for the CLI). Eligible Candle FlashAttention and Qwen3/Qwen3.5 RoPE CUDA routes activate automatically; their environment variables remain explicit rollback switches. Qwen3.8 is an independently gated family and must report its own execution path and evidence.
  • Linux/Windows source build for CUDA: build with cargo build --release -p izwi-cli -p izwi-server --features cuda, then run with --backend cuda or IZWI_BACKEND=cuda. The cuda wrapper includes Candle FlashAttention, while cudnn additionally enables matching Candle/cuDNN convolution paths.

See Also