How Voice AI Agents Work: An Architecture Guide for Software Engineers

The architecture at a glance

A successful call, end to end

Fictional call · happy path

A caller asks an equipment-rental service for a pickup reference. The agent retrieves it and reads it aloud. The caller asks for a text, the backend submits the message successfully, and the agent confirms the outcome before saying goodbye and closing the call.

Assume an authorized earlier step established identity and the permitted saved number. The example’s dialogue, data, and timing are illustrative; interruptions and failure cases come after the basic flow.

The caller experiences a conversation. Underneath it, audio, model responses, and backend work progress separately. The rest of the guide explains how those activities stay coordinated.

How audio and backend control meet

The architecture has two paths. Audio carries speech to the Realtime session and back to the caller. Control carries events and commands, such as a finished turn, a tool request or a new response request.

For this phone-call architecture, the backend attaches to the same Realtime session over a sideband, a separate server connection alongside the primary audio connection. It can observe events, update instructions, and return tool results while audio stays on the media path.

This is a supported connection pattern, rather than a requirement that all voice applications route audio the same way. OpenAI’s server-side controls guide describes the arrangement.

Three responsibilities make this work:

• Model: interprets speech and chooses a response, including whether to request a tool.
• Voice runtime: manages turn detection, streaming, and playback.
• Backend: supplies application context, executes permitted work, and coordinates the call’s lifecycle.

The provider, framework and application code may share these responsibilities. The important design choice is who owns each event and the action it triggers.

A speech-to-speech model brings audio understanding and speech generation into the model interaction. It still needs the application to retrieve private facts and perform business actions. In the rental call, the model can understand “text it to me,” but the messaging service remains the system that accepts the send.

Prepare the session and its context

Set up the session’s role, instructions and tools before the first exchange. Instructions can ask for brief replies, clear references and honest failure messages. Tool definitions describe the available operations and their arguments; the backend implements them.

The greeting also needs an explicit owner. Once the connection is ready, application code can request an opening response rather than leaving the caller waiting to discover who should speak first.

Meanwhile, the backend can load the small set of permitted facts likely to matter, such as whether a pickup reference exists and which saved contact may receive a text.

This early loading is prefetching. If it completes during the greeting and the caller’s first request, a read tool can use the cached result. A cache miss still needs an authorized lookup or a clear missing-information response.

Prefetch only facts the call is likely to need, and decide how long they remain fresh. Loading a reference early can save a lookup; sending a text still requires a separate action.

The prompt is only one part of the model’s context, the information available for its next response. On this call, context grows to include the caller’s request, the retrieved reference, the request for a text, and the tool’s result.

Instructions shape behavior; facts and outcomes give the model something reliable to talk about. Anthropic’s context engineering article explains why maintaining that information is broader than writing a prompt.

The backend also needs its own state. It should be able to distinguish a requested action, an action in progress, and a known or unknown outcome.

A conversational history may help the model remember the exchange, but application decisions need explicit evidence. For example, “I’ll send that” in the transcript cannot stand in for a result from the messaging service.

From caller speech to an agent reply

The caller asks, “What is my pickup reference?” In a turn-based Realtime interaction, the runtime identifies a caller turn and triggers, or allows the application to trigger, a response. Turn-based describes this organization of replies. It does not mean the microphone stops listening while the agent speaks; interruptions can still be supported.

Voice activity detection, or VAD, detects speech activity. Realtime’s silence-based mode uses a pause to identify a turn boundary; its semantic mode also estimates whether the caller has finished the utterance. That distinction matters when someone says “My reference is…” and pauses to find a piece of paper. The VAD guide describes these controls.

A shorter wait can make replies feel prompt while also ending a caller’s turn too early. A longer wait gives the caller room but may make every exchange feel slow. There is no single pause length that suits every task, speaking pattern, and acoustic environment. The choice has to be tested with representative calls.

Once the request reaches the model, a read tool supplies the reference. The backend returns the authorized value, and the model turns it into speech. In the timeline, it generates “R, four, M. Two, zero, eight.”

Speech streams toward the caller as the model generates it. Generation can finish while playback is still delivering the first few characters. Track both stages: the next reply, an interruption and the goodbye may depend on playback progress.

Execute a tool and deliver its result

After hearing the reference, the caller says, “Please text it to my saved number.” The model recognizes an action request and proposes a tool call. A short acknowledgment can make the work understandable to the caller while the backend checks and executes the request.

The contract below is illustrative, not runnable API code. It shows the information crossing the boundary. The backend resolves the reference and permitted destination from call state, so the model does not select an arbitrary phone number.

Model input
  Caller requested a text.
  Pickup reference is available.

Available tool
  name: send_pickup_details
  purpose: Send pickup details to the permitted saved contact.
  arguments: {}  (backend resolves the details and destination)

Model → backend
  request_id: t2
  tool: send_pickup_details
  arguments: {}

Backend execution
  Check permissions, destination, and action state.
  Submit the request to the messaging service.

Backend → model
  request_id: t2
  outcome: accepted_for_sending
  delivery: not_confirmed

Spoken reply after earlier playback finishes
  “The text is on its way. Anything else?”

The matching identifier associates the result with the correct request. Returning it adds evidence to the model’s conversation context; a subsequent response can then explain the outcome. OpenAI Realtime exposes this through function-call items and correlated function results. The function-calling guide documents that exchange.

Notice the timing in the happy path. The backend starts the send while the agent’s acknowledgment is playing. If the result arrives quickly, it can be returned to the model immediately.

Requesting the next spoken response is a separate decision: this example waits for the earlier playback to finish so the two replies do not overlap.

That is one coordination policy, not a universal Realtime requirement. Its purpose is to let useful backend work overlap with speech while preserving a clear conversation. A bounded wait and recovery policy are still needed if an expected playback signal never arrives.

Here, the service accepted the text for sending. The reply can say it was submitted; delivery to the handset is still unconfirmed. A failure or unknown result needs its own honest response.

Handle interruptions, pacing, and silence

Now change one moment in the successful call. While the reference plays, the caller says, “Could you repeat that more slowly?” The goal stays the same, but the runtime must stop the old answer before delivering the repeat.

An interruption requires the runtime to:

• Stop obsolete playback.
• Cancel ongoing generation where applicable.
• Keep conversation context consistent with the audio delivered.

Otherwise, the agent may talk over the correction or assume the caller heard words still waiting in the output queue.

In Realtime, responsibility for truncating unplayed audio differs between server-managed SIP/WebRTC playback and client-managed WebSocket playback. The conversation guide explains those transport differences.

Then the model has to interpret the new intent. “Say it slowly” calls for a repeat; “Actually, I need something else” changes the task. A cough or a brief “mm-hmm” may not request a turn at all. LiveKit’s interruption guidance illustrates why detecting speech and deciding whether it should interrupt are separate concerns.

Pacing can prevent some of that friction. Short replies leave room for the caller, and important identifiers may need slower delivery or grouping. These are useful prompt instructions, but their effect must be evaluated in actual audio.

Realtime prompting guidance covers tone, length, and pacing. Faster generation alone cannot make a reference easier to write down.

Silence also needs context:

• During a caller’s sentence: a pause helps determine whether the turn has ended.
• After the agent finishes: the caller may be thinking or writing.
• While a tool runs: the application still owes the caller an answer.
• After an expected playback notification fails to arrive: the runtime has a coordination problem.

Treating all four as “the caller is inactive” produces the wrong response.

A backend can use those states to decide when a check-in is appropriate and explicitly tell the model that no new caller speech has arrived. Silence should not become an invented request or consent. The aim is to give the caller room while keeping unexplained waiting bounded.

Close the call and handle uncertainty

The happy path ends with “Thanks, that’s all.” The model can request an end-call action, but the backend still has to coordinate the farewell and the actual disconnection. A spoken goodbye alone leaves that lifecycle unfinished.

The example requests a closing response, waits for its playback, then asks telephony to hang up. Two completion signals matter:

• Generation complete: OpenAI’s response.done reports that a response has finished generating.
• Playback complete: for SIP/WebRTC, output_audio_buffer.stopped reports that the server output buffer has drained.

Neither proves exactly what a person heard at the other end. The event reference defines those meanings.

After requesting hangup, wait for evidence that the line closed. That is the final timeline checkpoint. Set a recovery policy for missing completion signals and decide what to do if the caller speaks during the farewell.

Uncertainty can arise earlier, too. Suppose the messaging service accepts the send but its response is lost. The backend sees a timeout without knowing whether the action occurred. Repeating the request blindly may send a duplicate.

An idempotency key can identify repeated attempts as the same logical operation when the receiving service supports that contract; otherwise, the backend may need to check the outcome before deciding what to do. AWS’s guide to safe retries explains this boundary.

An interruption does not resolve that uncertainty or undo an action already accepted elsewhere. Canceling speech, canceling pending work, and reversing an external effect are different operations.

The agent should communicate only what the system knows, with an appropriate handoff or fallback when it cannot continue. Recovery for a model or backend outage must also be reachable when that component is unavailable.

Observe and evaluate the whole call

If a caller reports “It said it would text me, then went quiet,” the architecture gives you a way to investigate. Did the model request the tool? Did the backend run it? Did the result return? Was the next reply waiting for playback, or was it generated and never delivered?

A useful trace connects the caller turn, model response, tool request, service outcome, playback events, and call closure. Correlation identifiers and timestamps let you follow the exchange across systems. Each view contributes different evidence:

• Transcript: helps explain intent.
• Audio: reveals pacing and overlap.
• Backend events: show decisions and observed outcomes.

None of these views replaces all the others. Missing evidence should remain a visible gap.

Measure each wait separately: connection to greeting, end of the caller’s turn to first audible reply, tool execution and interruption recovery. These timings help separate a slow lookup from a late turn boundary or stalled playback. Restrict access to recordings and transcripts, and set retention limits for caller data.

The same happy path becomes a useful test case. First verify that the reference is correct, the permitted text request executes, the spoken reply matches its result, and the line closes.

Then introduce a delayed tool, an interrupted answer, caller silence, or a missing completion event. This tests the relationships between components, including failures that a plausible transcript can hide.

Start by making one complete call observable. Then test the moments where activities overlap: a tool returns during playback, the caller interrupts, or the line must close. Those transitions show whether the architecture can keep the conversation and backend work in sync.