Testing Agent Interfaces with Golden Workflows

An agent-interface test should begin with an observable product outcome and then verify every boundary required to reach it. Schema checks alone miss behavioral failure; prompt evaluations alone miss protocol, permission, and state defects.

Define a golden workflow

The customer request lab’s primary workflow is:

Find an untriaged customer email.
Inspect the selected CRM record and applicable handling rule.
Explain the facts and recommend the correct queue and status.
Preview the CRM change.
Require explicit human confirmation.
Apply exactly once and verify the resulting state.

The final assertion is not “the model said success.” It is “the case is at version 2 with the expected status and assignment, and one audit record connects the proposal, confirmation, and state transition.”

Test five layers

| Layer | Representative assertions | | --- | --- | | Protocol | Initialize, list tools, call tools, read UI resource | | Contract | Input and structured output validate against declared schemas | | Decision | Correct tool selected; dangerous or irrelevant tool avoided | | Authority | Audience, scope, confirmation, replay, and tenant isolation enforced | | Operation | Latency, error rate, recovery, final state, and evidence acceptable |

These tests can share fixtures and reporting, but they should retain distinct failure labels. A schema regression has a different owner and repair path than a model-selection regression.

Make negative cases mandatory

The useful suite asks what must not happen:

wrong audience       -> 401, no mutation
read-only scope      -> preview/apply denied, no mutation
missing confirmation -> 409, no mutation
stale case version   -> 409, new preview required
replayed capability  -> rejected unless same completed idempotency key
injected case note   -> remains data, cannot expand authority
expired tenant       -> 401, records purged

Seed these conditions intentionally. A release gate that has never observed itself catching a broken boundary is mostly a hope.

Separate deterministic and model-dependent tests

Contract validation, OAuth behavior, tenant isolation, version comparisons, idempotency, and final state should be deterministic.

Tool selection, explanation quality, and correct avoidance depend on the host, model, system instructions, conversation, and tool catalog. Run those against named configurations and record them in the report. Do not hide a model change inside a generic “agent” label.

Version the manifest

An internal test manifest should make the release expectation reviewable:

{
  "manifestVersion": "0.1.0",
  "protocolVersion": "2026-07-28",
  "endpoint": "/mcp/ops-demo",
  "assertions": {
    "contract": ["five_tools", "valid_structured_output"],
    "permission": ["wrong_audience_denied", "read_scope_cannot_write"],
    "confirmation": ["missing_denied", "replay_denied", "stale_denied"],
    "latency": { "p95Ms": 1200 }
  }
}

Store the manifest with the implementation. A change to a tool schema, scope, confirmation rule, or expected outcome becomes a visible versioned decision.

Produce evidence, not only a pass badge

A useful release report contains:

the endpoint and protocol version tested,
client, model, and authentication profile where relevant,
tool catalog and schema diff,
each assertion and its evidence,
latency and error distributions,
final product-state verification,
comparison with the last accepted run,
and a release verdict with acknowledged exceptions.

“Pass” should be reproducible. If a reviewer cannot see which request, response, and final state support it, the verdict is too opaque.

Use production evidence carefully

After release, monitor compatibility and actual outcomes. Redact credentials and sensitive fields. Sample failures for diagnosis. Connect tool traces to reversible changes and user corrections without turning observability into unrestricted content retention.

The evaluation set should evolve when real use reveals a new failure class. It should not change merely to make a release pass.

Claim boundary

A passing suite supports a bounded release decision for the tested contracts, clients, workflows, and risk assumptions. It does not establish universal safety or permanent compatibility.