Demos Are Easy. Systems Are Not.

An AI demo is a possibility argument. A production system is a responsibility.

The demo asks whether a model can do the interesting part once. The system has to do the whole job under ordinary conditions, preserve evidence, recover from failure, and make its limits clear to the people depending on it.

Confusing those two things is the most common mistake in applied AI.

Why the demo works

A good demonstration has several hidden advantages. The input is chosen because it is representative and legible. The operator knows what the model should do. The environment is quiet. Nothing upstream is late. No permission changed overnight. The API is available. A person is watching closely and can repair a bad prompt or rerun the step.

These are reasonable conditions for learning. They are not operating conditions.

Real work arrives incomplete, duplicated, contradictory, mistimed, and attached to consequences. A customer sends a document format the test set never included. A field changes meaning after a system migration. A tool succeeds but the record is not committed. The model returns a plausible answer without the one source the reviewer would consider authoritative.

The production problem begins exactly where the demo ends.

The system around the response

For a model-dependent feature, the complete system usually contains at least these layers:

| Layer | Question it must answer | | --- | --- | | Experience | What job is the person trying to complete? | | Context | What does the model need to know for this case? | | Retrieval | Which source material is relevant and allowed? | | Tools | What can the system read, calculate, or change? | | State | What has happened, and what remains incomplete? | | Model | Which judgment or generation requires learned capability? | | Validation | Is the output structurally and operationally acceptable? | | Evaluation | Did the full task succeed on representative cases? | | Control | What needs approval, a limit, a fallback, or a stop? | | Operations | Who sees failure and what can they do about it? |

The model is one row. It may be the row that makes the product newly possible, but it does not absolve the rest.

Completion is not a sentence

Agents introduce a particularly dangerous failure mode: they can say they completed work that did not complete.

Suppose an agent is asked to update a customer record and send a confirmation. It calls the update tool, receives an ambiguous response, drafts a confident summary, and sends the email. The trace is green because no request threw an exception. The task still failed.

Completion has to be verified in the system that owns the state. Read the record back. Compare the expected change. Confirm the message identifier. Preserve the evidence. If verification fails, the agent should retry only when the action is idempotent and the failure is understood. Otherwise it should stop and route the case.

This is ordinary distributed-systems work. AI did not repeal it.

Evaluation before confidence

Teams often postpone evaluation until the system is nearly finished. That reverses the useful order.

Create representative cases before architecture hardens. Include normal cases, rare but consequential cases, malformed inputs, conflicting sources, missing permissions, and tool failures. For each case, define what evidence an experienced reviewer would use.

Then measure layers separately:

Did retrieval find the right material?
Did the model use the material correctly?
Did the tool call contain the right arguments?
Did the underlying system change as expected?
Did the user get a legible result?
Was the cost and latency acceptable?
Did the system stop when it should?

A single score can hide the reason a system fails. Layered evaluation points to the repair.

The NIST AI Risk Management Framework emphasizes governing, mapping, measuring, and managing AI risk as an ongoing process. The language is broader than a product team’s test harness, but the practical implication is the same: evaluation is not a launch gate you pass once. It is part of operation.

Failure is a product surface

Ordinary software has error states. AI software has error states plus uncertain success states.

Design both.

If the system cannot find enough evidence, say what is missing. If two sources conflict, show the conflict. If a tool is unavailable, preserve the work already done and explain what remains. If a case is outside the evaluated boundary, route it without forcing an answer.

Users will learn the system’s actual reliability from these moments. A graceful refusal builds more trust than a polished hallucination.

Cost and latency are architecture

The demo might make one expensive model call. The production workflow may call models thirty times, retrieve thousands of tokens, invoke tools, retry failures, and hold a user waiting.

That multiplier changes the design. Use deterministic code for deterministic work. Cache stable context. Route simple cases to smaller models. Set budgets at the task level. Make long work asynchronous. Report progress from durable state rather than keeping a browser request alive.

OpenAI’s discussion of its agentic harness and repeated work describes this compounding effect: a small cost inside a repeated model and tool loop is paid many times. Production efficiency comes from the harness as much as the model.

Ownership is the final layer

Every production AI system needs a person or team responsible for:

source access and data changes,
tool permissions,
evaluation cases and release thresholds,
incidents and user reports,
model or prompt changes,
cost and latency budgets,
and the decision to narrow or disable behavior.

Without that owner, quality decays quietly. The model changes. The corpus grows stale. Users adapt the workflow. The evaluation stops representing real use.

A production-readiness test

Before widening access, ask:

Can we state the job and completion condition without mentioning the model?
Do we know which inputs are authoritative?
Are tool permissions narrower than the user’s broad account access?
Can every consequential action be verified?
Do we have cases that represent normal and dangerous failure?
Can an operator understand and recover a failed run?
Can we change the model without losing the definition of quality?
Does someone own the answer to all seven questions?

If not, the demo may still be valuable. It has shown that the interesting part is possible. Now the system work can begin.

What hardening looks like in a real workflow

Consider a document-review prototype. A user uploads a contract, asks several questions, and receives fluent answers with citations. The demonstration is useful: it proves the model can work with the material and helps the team see a possible experience.

Production begins when the team asks what happens around that moment.

Where did the document come from? Does the uploader have the right to use it? Is the file a new agreement, an amendment, or a duplicate? Which pages failed OCR? What should happen when two versions conflict? Can one user’s document appear in another user’s retrieval context? How is the document removed from every index when retention expires? Does a citation point to the exact supporting language or merely a nearby page? Can the reviewer correct an extracted date without silently rewriting the source?

None of those questions diminishes the prototype. They reveal the system the prototype was pointing toward.

A sensible hardening sequence might look like this:

Assemble representative documents, including scans, tables, amendments, duplicates, and malformed files.
Define the outputs the workflow actually needs and the evidence required for each.
Separate deterministic parsing and validation from model-dependent interpretation.
Enforce identity and access during retrieval, not after an answer has been generated.
Build a review interface around uncertainty and consequential fields.
Measure field quality, evidence quality, correction effort, latency, and cost.
Add retries and fallbacks that do not duplicate downstream actions.
Release to a small group whose corrections become new evaluation cases.

This sequence is less dramatic than a live demo. It is also where confidence is earned.

Keep the prototype, but name it correctly

The answer is not to stop prototyping. Fast, imperfect prototypes are one of the best ways to learn whether a model capability belongs in a workflow. They make abstract conversations concrete and allow the team to discover that its first idea was wrong while change is still inexpensive.

The mistake is allowing the prototype to carry claims it has not earned. A prototype can demonstrate possibility. It cannot establish reliability without representative cases. It cannot establish operating cost from three hand-picked runs. It cannot establish security when every tester shares the same account. It cannot establish adoption while the builder stands beside the user explaining each step.

Naming the stage accurately improves decision-making. Teams can celebrate a strong prototype while remaining clear about the open work. A useful review asks:

What did we learn about the user and the job?
Which model behavior appears promising?
Which failures would matter in production?
What assumptions did the demonstration hide?
What evidence would justify the next investment?

That posture preserves momentum without turning excitement into certainty. It also protects the people who will eventually operate the system. They deserve more than a polished handoff and a list of prompt tips. They need an application with known boundaries, visible behavior, and a practical way to improve it.