Adding a large language model (LLM) changes how you design a feature. A support chatbot can return a fluent answer based on an old policy. Every API call may succeed, yet the product still gives the wrong answer.

Treat the model as one component in your application. It can interpret a request or propose a next step. Your code still controls permissions, stores state and handles failures.

These six mental models connect familiar engineering decisions to the new sources of uncertainty. We’ll move from what the model may do and what it sees to how the system stores progress, manages timing and tests behavior.

1. Model boundaries: keep application rules in code

Imagine a support product that accepts either typed questions or calls. It might answer a policy question, collect missing information, or route a request for human review.

Starting with “Which model should we use?” leaves most of that system unspecified. Start instead with the decisions the product must make.

A model can interpret “I was charged twice” as a possible billing issue. That is useful judgment over ambiguous language.

The application still authenticates the customer, restricts which records are visible, and decides which operations may run. If refunds above a defined amount require review, encode that rule where the action is authorized. A prompt can describe the rule; it cannot replace the enforcement.

This resembles designing an interface to an external service, with an additional distinction: valid structure is not valid meaning. A response can match a schema while selecting the wrong customer request.

A schema describes the fields; your application checks whether the proposed values and action make sense in the current context. When the choice is ambiguous, asking a question may be the correct outcome.

Begin with a fixed workflow when the legal paths are known. Letting the model choose successive tools can help with open-ended investigation, but introduces more paths, latency and recovery decisions.

Put bounds on those choices and measure whether the flexibility helps. A single model call can be enough; multiple agents are not a prerequisite for an AI product.

Once you decide which choices belong to the model, you need to decide what information it receives. That input can change its behavior without any change to your application code.

2. Context: inspect what the model receives

In ordinary software, changing a configuration file has no effect if the running process reads a different one. Prompt changes have the same problem.

A repository’s system instruction may be only one contributor to the model request:

  • Middleware can add instructions.
  • Tool descriptions can guide operation choices.
  • Retrieved passages can supply facts.
  • Previous messages can shape the interpretation of a new request.

Context is the information supplied to one model call. Trace how it is assembled: which code adds instructions, which documents supply facts, and which version reaches the model. Record enough detail to investigate a bad answer.

Protect sensitive data in those records. If you edit a prompt the running system never loads, the fix cannot change its behavior.

Anthropic’s context-engineering article describes this broader task of selecting the information available at inference time. The practical design choice is which facts and instructions belong in this call—not how much text you can fit into it.

Consider a knowledge chatbot answering “How long do I have to return this item?” In a fictional policy set, an old document says 30 days and the current applicable policy says 14.

Retrieval-augmented generation (RAG) means finding relevant material and supplying it to the model so it can answer from that evidence. It combines a search problem with an answer-generation problem; it does not give the model permanent, automatically correct memory. Anthropic’s retrieval primer describes the retrieval-to-context pipeline.

Illustrative system sequence

An answer inherits the evidence it receives

Old policy

  1. Retrieve — Search ranks the old policy first. No generation failure is needed to produce the wrong current answer.
    Search: v1 selected · Context: v1: 30 days · Model: Waiting · Reader: Waiting
  2. Assemble — The context builder places that passage in the model request.
    Search: v1 selected · Context: Sent v1 · Model: Reads 30 days · Reader: Waiting
  3. Answer — The answer can cite its source and still use the wrong policy version.
    Search: v1 selected · Context: Sent v1 · Model: Cites v1 · Reader: Wrong policy

Current policy

  1. Retrieve — In this fictional product, an effective-date filter selects the currently applicable policy.
    Search: v2 selected · Context: v2: 14 days · Model: Waiting · Reader: Waiting
  2. Assemble — The input now contains the evidence needed for the intended answer.
    Search: v2 selected · Context: Sent v2 · Model: Reads 14 days · Reader: Waiting
  3. Answer — The search and answer checks pass for this illustrative case; this is not a general accuracy guarantee.
    Search: v2 selected · Context: Sent v2 · Model: Cites v2 · Reader: Current answer

If the chatbot received only the obsolete passage, investigate document lifecycle, filtering and ranking. If it received the correct passage but invented an exception, investigate how the answer was produced and checked.

A citation alone does not settle either question: the cited passage must support the claim and apply to the user’s situation.

That separation changes your tests:

  • Search: does the necessary evidence appear among the selected results?
  • Generation: does the answer follow that evidence and admit missing information?

Extra text may supply a missing fact, but it can also distract the model and add cost or delay. Test that trade-off on real tasks. Once a call has enough context, the next question is how to keep its useful results across steps.

3. State: give each update an owner

Context is an input to computation. It is not a substitute for application state. A document-processing workflow makes the distinction visible: one component extracts an invoice identifier, another decides whether review is needed, and the UI shows progress.

Each step may use a model, ordinary code, or both. The workflow still needs an unambiguous rule for combining their updates.

Suppose the stored object is {invoice_id: "A7"}. A review step returns {review: "required"}. Replacing the entire object loses the identifier. Merging only the field owned by the review step retains it. Both operations are deterministic; no better prompt will fix the wrong update operation.

Illustrative system sequence

A partial update needs an explicit merge rule

Replace object

  1. Extract — The extractor returns an invoice identifier; the application stores it.
    Extractor: ID: A7 · Reviewer: Waiting · State: ID: A7 · UI: Empty
  2. Review — Replacing the whole object with this partial update discards the identifier.
    Extractor: ID: A7 · Reviewer: Required · State: ID lost · UI: Empty
  3. Display — The UI receives a valid-looking object with incomplete state. Changing the extraction prompt would not repair the merge bug.
    Extractor: ID: A7 · Reviewer: Required · State: ID lost · UI: Missing ID

Merge owned fields

  1. Extract — The extractor owns the invoice identifier field.
    Extractor: ID: A7 · Reviewer: Waiting · State: ID: A7 · UI: Empty
  2. Review — A field-aware update retains the identifier while adding the review flag.
    Extractor: ID: A7 · Reviewer: Required · State: ID + review · UI: Empty
  3. Display — This solves disjoint-field updates. Two writers changing the same field still need a conflict policy.
    Extractor: ID: A7 · Reviewer: Required · State: ID + review · UI: Both fields

A reducer is the function that decides how an incoming update changes existing state. A checkpoint is a saved workflow snapshot used to resume execution.

For example, LangGraph documents per-field reducers; that is one implementation of the idea, not a requirement to use that framework. An explicit state object and ordinary application code can express the same ownership decision.

The update rule follows from what each field means:

  • Event list: append new events, accounting for event identity so replay does not create duplicates.
  • Status field: replace an older value when the new state should supersede it.
  • Shared field: define an ordering or conflict policy when two writers can change it.

Keep three roles clear:

  • Model context: what a particular call sees.
  • Workflow state: what the application knows about progress.
  • Business state: the durable record of what actually changed.

Each can have an authoritative owner without sharing one giant object. A checkpoint can preserve a proposed booking; it does not itself create a reservation.

When debugging, follow the field: who writes it, how the update is merged, and which saved value survives a retry. This gives the team a concrete path to inspect when the UI loses progress.

4. Timing: track what the user experiences

State is not only about stored fields. In a voice product, it includes what is happening right now: the caller is speaking, audio is queued, playback is active, a tool is pending, or the call has ended.

This should feel familiar if you have built streaming interfaces or asynchronous jobs. Completion at one stage is not completion at the next.

Consider a voice assistant ending a call. The model can finish generating a farewell while audio is still being played. It can also request an end-call operation before the caller has heard the farewell.

If the backend disconnects as soon as it receives that request, the generated transcript may look perfect while the caller hears only “Good…”

Illustrative system sequence

Generation, playback and disconnection are different events

Disconnect early

  1. Request — The caller signals that the conversation can end.
    Caller: Finished · Audio: Idle · Model: Responding · Backend: Connected
  2. Generate — The model finishes producing the farewell while playback is still in progress.
    Caller: Waiting · Audio: Playing · Model: Generated · Backend: Connected
  3. Ask to end — A tool request tells the backend what action is requested; it does not mean the call has ended.
    Caller: Hears “Good…” · Audio: Playing · Model: End requested · Backend: End pending
  4. Disconnect — This fictional backend disconnects immediately, cutting the farewell short.
    Caller: Cut off · Audio: Interrupted · Model: Finished · Backend: Disconnected

Wait for playback

  1. Request — The same request starts the corrected sequence.
    Caller: Finished · Audio: Idle · Model: Responding · Backend: Connected
  2. Generate — Generation finishes; playback remains active.
    Caller: Waiting · Audio: Playing · Model: Generated · Backend: Connected
  3. Ask to end — The backend holds the requested terminal action pending the playback condition.
    Caller: Listening · Audio: Playing · Model: End requested · Backend: End pending
  4. Play — In this idealized path the farewell is heard before disconnection.
    Caller: Heard farewell · Audio: Drained · Model: Finished · Backend: End pending
  5. Acknowledge — A transport-appropriate playback signal releases the gate. It is not proof of acoustic delivery through every downstream device.
    Caller: Heard farewell · Audio: Drained · Model: Finished · Backend: Can disconnect
  6. Disconnect — Only now does the backend perform the terminal operation. Interrupted playback and missing signals need a separate policy.
    Caller: Call ended · Audio: Drained · Model: Finished · Backend: Disconnected

The backend must wait for the relevant playback signal before it hangs up. The exact signal depends on the audio and phone system.

A buffer-completion event is evidence about that buffer, not independent proof that every downstream device delivered sound. The animation assumes an uncomplicated delivery path so the timing relationship is visible.

Interruption complicates the dependency. If the caller speaks during the farewell, draining the old audio may no longer be desirable. Decide whether to cancel the pending hangup and resume the conversation.

If playback fails or its completion signal never arrives, use a bounded timeout and a defined fallback rather than waiting forever. These are lifecycle policies your runtime must own.

Ask which stage has finished and what may happen next. A generated chat reply does not mean a backend action has finished. In voice, speech generation, playback and disconnection each have their own progress.

Evaluate the experience with aligned events and, when needed, audio; a transcript alone cannot show every timing failure.

5. Evaluation: test the whole workflow

You now have several plausible failure locations: the model’s interpretation, the evidence it received, a state update, or an event-ordering rule. Logs and service-health metrics still matter.

A trace records the inputs, tool calls, results and state changes across a request. It helps you find where behavior went wrong. It does not reveal the model’s internal reasoning.

An evaluation runs representative cases and checks whether the resulting behavior meets explicit expectations. Keep unit and integration tests for permissions, merge rules and timeouts.

Add cases that vary language and context, and repeat trials where output variation matters. An answer can be phrased differently and still be correct; a required authorization step cannot simply disappear.

Different questions need different checks:

  • Mechanical checks: did the invoice ID survive, was the policy version correct, and did disconnect follow the expected playback event?
  • Judgment-based checks: was a clarification useful, and did the answer overstate the evidence?

A model can help grade answers at scale. First test its rubric, the criteria it uses, against examples that people have reviewed. Inspect disagreements before trusting its scores.

Anthropic’s evaluation guide distinguishes grading methods, repeated trials and the final environment outcome.

Suppose several chatbot failures all contain the old policy. A useful hypothesis is that the retriever lacks an effective-date filter. Test that change against the failed cases and against questions that legitimately ask about a historical policy.

Adding “always be accurate” to the prompt does not test the hypothesis. Filtering out all old documents may fix one group while breaking historical questions.

Group similar failures before changing a prompt so the fix addresses a pattern. For a clear permission bypass or state-update bug, repair the known defect directly.

6. Scope: build around proven behavior

These mental models finally meet in scope. “Build a voice support agent” could mean reading a known answer, handling ambiguous requests, coordinating tools, surviving interruptions and completing a transfer. A short demo of the first capability says little about readiness for all the others.

Define the product’s capability envelope: which tasks it has passed under which conditions, and what it does when a request falls outside those limits.

For a first knowledge chatbot, that might mean answering questions supported by current policy documents and handing off unsupported cases. It need not include changing accounts. For a voice pilot, clear short exchanges may be the initial scope, while tool-heavy conversations remain under evaluation.

Apply the same care you use for a software release. Decide what counts as successful handling, including a useful clarification, refusal or handoff. Set error limits that reflect the harm a mistake could cause.

Track behavioral results alongside latency and availability. Revisit the envelope as prompts, models, documents and user traffic change; a fixed application binary does not imply every input to its behavior stayed fixed.

Start with one small task that works from input to outcome. Give the model the facts it needs, make state changes explicit, and test the resulting behavior before adding more tools or tasks.

Building an AI product still draws on service design, state management and release engineering. The added work is to make model behavior visible and testable, so each design decision rests on evidence.