A voice conversation in Izwi is run by an agent: it listens, decides what to say, and can look things up or act before it answers. This guide covers what you can give that agent, in the app and over the API. Everything runs on your own machine. For starting a conversation and choosing an engine, see Voice Mode.

What an agent can do

Tools need a model that can call them. The Standard engine can, with its Qwen3.5-4B assistant. Of the Compact engine’s models, LFM3-Audio-2B-GGUF can and LFM2.5-Audio-1.5B-GGUF cannot. Agents and tasks need the Standard engine.

Tools

Open Voice settings › Tools.

Date and time

Turn on Date and time and the assistant can answer “what’s the date?” or “what time is it?” from the real clock. It is off until you turn it on, on both engines. On the Standard engine, offering any tool makes the assistant read a longer prompt. With the Qwen3.5 assistant Izwi reuses the part it has read before, so on an M1 Pro the cost is well under a second a reply, where it used to be about two; the first reply after Izwi starts still pays it in full. A reply that calls a tool takes about as long as one that does not. On the Compact engine, LFM3 Audio tends to call the tool when you only say hello, and then opens with the date and time. You rarely need it there: Izwi tells the Compact engine the date and time at the start of every turn, so it can answer “what time is it?” with the tool off.

Tool servers

A tool server is a program that offers tools over the Model Context Protocol (MCP): a stock lookup, a calendar, a ticket system. Add its name and address, and an access token if it needs one. The assistant can then call its tools, and Izwi runs the calls. Izwi’s server makes the connection, not your browser. For safety it only connects to hosts you have allowed. Set IZWI_MCP_ALLOWED_HOSTS to a comma-separated list of hosts (mcp.example.com, 127.0.0.1:9000, or * for any) before starting Izwi. Until it is set, a conversation that names a tool server by address fails to start and says so. The access token is kept in this device’s browser storage and sent to Izwi when a conversation starts.

Tool servers that run as a program

Many MCP servers are not web services but programs you run: a file browser, a database reader, a calendar bridge. Izwi starts these itself. Because a program can do anything your account can, only a file on the computer Izwi runs on can name one; nothing typed into the app or sent over the API can. Create mcp.json in Izwi’s data folder, next to izwi.sqlite3: ~/Library/Application Support/izwi on macOS, ~/.local/share/izwi on Linux, %LOCALAPPDATA%\izwi on Windows (IZWI_MCP_SERVERS_FILE names another place). It takes the same form other MCP apps use:
Restart Izwi. Each server then appears in Voice settings › Tools under Set up on this server, with a switch. A remote server listed in this file needs no entry in IZWI_MCP_ALLOWED_HOSTS, and its token stays in the file.
  • Give the full path to the command. The desktop app does not see the PATH your terminal has, so npx or uvx alone is often not found. env adds environment variables and cwd sets the folder the program starts in.
  • One copy runs for everyone. A program starts the first time a conversation uses it, is shared by every conversation after, and is started again if it exits. Its first start may take a while if it has to install itself; Izwi waits up to a minute.
  • Anyone who can use this Izwi server can use these tools. List only servers you would let every caller of the API reach.

What you see

When the assistant uses a tool, the transcript shows a short line ahead of its reply: “Using stock level…”, then “Used stock level”. A tool that fails or takes too long says so, and the assistant tells you it could not check. If a tool keeps the assistant silent for more than a second and a half, it says “One moment.” first.

Agents

One assistant with one set of instructions suits most conversations. Agents help when a conversation has distinct jobs: a front desk that greets and routes, a billing agent that handles refunds. Open Voice settings › Agents and add at least two. The first agent in the list answers first. When an agent decides the conversation belongs to another, it hands over; the transcript shows “Handed over to Billing”, and that agent answers from then on, until it hands back or the conversation ends. A small model will sometimes say “I’ll transfer you” without doing it, or ask again for something you already said. Clear, short instructions (“For anything about invoices, hand over to billing at once”) help.

Turn taking

Open Voice settings, under Conversation. Izwi decides you have finished from silence alone. It does not use a separate model to guess whether a sentence is complete.

For developers

The voice WebSocket (/v1/voice/realtime/ws) takes all of this in input_stream_start. A session with the date and time tool, one tool of your own, and a tool server:
An mcp_servers entry without a url names a server from mcp.json; GET /v1/voice/tool-servers lists them. When the reply calls get_weather, the server sends function_call with a call_id. Your client runs it and answers:
A tool marked "background": true does not hold the reply: the assistant says it has started, and the output you send later is told on the next turn. Two agents, the second with a task to collect:
The server sends agent_handoff when the conversation moves, and task_completed with the collected fields once the agent has them all. The turn-taking switches are resume_false_interruptions, adaptive_silence and preemptive_generation. A client that sets resume_false_interruptions must stop playback on turn_paused without discarding what it has queued, and play on at turn_resumed. Clients that speak the OpenAI Realtime protocol (/v1/realtime) get the same agent behind it: function tools as usual, and MCP servers as {"type": "mcp", "server_label", "server_url"} entries in tools. There a tool call waits for your client’s approval unless the entry says "require_approval": "never": the response ends with an mcp_approval_request item, and an mcp_approval_response item runs the call or declines it. The API reference lists every field and event. To test an agent before people talk to it, scripts/bench/run-voice-agent-eval.py plays a caller against it and scores the calls.

Limits

  • On the Standard engine a conversation remembers what its tools found, so a later question can build on an earlier lookup without calling the tool again. A long result is kept in shortened form (about 1,500 characters). On the Compact engine tool calls and their results last for one turn.
  • A tool server named by address must be allowed in IZWI_MCP_ALLOWED_HOSTS; one that runs as a program must be in mcp.json. A voice conversation does not ask you before a tool runs.
  • Agents and tasks are not available with the Compact engine. Tasks can only be set over the API.
  • A hum or a cough resumes the reply only where Izwi can tell it had no words. A short “yes” or “okay” while Izwi is speaking still interrupts it.