AI Agent Testing Framework: How We Test an Agent Before It Touches a Customer
How do you test an AI agent before it goes live, and which checks should be hard code gates rather than prompt judgment? A practical answer from agents we have shipped.

A solid ai agent testing framework has two layers. Code gates check facts that must never ship wrong: prices, links, categories, permissions, counts and whether anything can publish or send. Prompt-level evals judge tone and fit. Then you run shadow mode on real traffic before granting autonomy.
That split sounds tidy. Getting it right took us a few expensive mistakes, and the rest of this post is what those mistakes taught us about ai agent testing: where to put the hard checks, what to leave to the model, and how to plan the work so a launch is boring.
The short answer, expanded
Most teams start by asking whether the model's answers are good. That is the wrong first question. The first question is: what is the worst thing this agent can do with the tools I gave it, and can any code path do it at all?
On our MEC Artworks product-page agent, Claude vision turns product photos into store listings: copy, category, tags, attributes, alt text and SEO fields. Output lands as a hidden draft, and a person reviews and publishes. We did not test whether the model would behave and decide to publish. We wrote a test that fails the build if any code could set a product live. The agent's good manners are irrelevant to that check. It is mechanical, and it has to stay that way.
The same agent shows the other half of the rule. Code sets the price from size and technique within a band, links come from an allowlist, and categories come from the store's real list. Facts live in code. Prompts carry judgment: how to describe a piece, what to emphasize, how long the copy runs. Our position is simple: prompts carry judgment, code carries facts.
What changes the answer
There is no single recipe for testing LLM agents, because the right gate depends on the blast radius. Three variables move the answer most.
Reversibility of the action
A draft in a CMS can be deleted. An email to a customer cannot be unsent. Our own lead-acknowledgment agent sends without review, and that is defensible only because the gates around it are mechanical: a rule-based spam filter runs first, code (not AI) picks the case study, and sending stops on booking, unsubscribe, bounce, or a won or lost deal. Claude Haiku only writes the 130 to 200 word body. Everything irreversible is guarded by code you can unit test.
Anything customer-facing and irreversible without that kind of mechanical gate stays a draft. Full stop.
How the agent talks to other systems
Free text should never feed another system. We force the model to call one strict tool or answer against a JSON schema, and we put the client's real categories and tags into the schema as allowed values. This turns a fuzzy evaluation problem into a schema validation problem, which is cheap and deterministic.
There is a gotcha here that no eval suite would have told us about. Strict tool schemas reject minItems and maxItems. We added them, and every production call broke. Counts are now checked in code. Test the schema against the real provider endpoint, not only against a local validator.
Who the agent acts as
Permissions are a code gate, never a prompt instruction. Our Groohve MCP server exposes 113 tools to Claude and ChatGPT. Each request mints a 5-minute token for that user, so row-level security scopes every tool, and another customer's record returns "not found". Destructive tools need explicit confirmation. Even so, a live smoke test after launch found the permission check locked out most real owners. We fixed it the same day. The lesson: test with real accounts, not just a test user with every role.
Which checks should be code, which should be prompt
Here is how we sort them. If a wrong answer is a fact error that a customer or regulator could catch, it is a code gate. If a wrong answer is merely a worse answer, it belongs in the prompt and in an eval.
- Code gates: prices, URLs, category and tag values, permissions, item counts, rate limits, spend caps, whether output can reach a live state, idempotency of sends.
- Prompt judgment: tone, which detail to lead with, how to rank fit, how to phrase a refusal, what to ask next.
- Both, in sequence: the model proposes, code validates, and on rejection the retry sends the model the reasons it was rejected.
That last item matters more than it looks. Early on, our retry loop just reran the call. All three attempts failed the same way. Feeding the rejection reasons back fixed it. Also treat running out of tokens as a failure, never a partial result. A truncated output that passes a lenient check is how bad data gets published.
A related trap: examples inside prompts. One example size in a prompt showed up on almost every product the agent described. We now write principles, not examples, and test for repeated values across a batch of outputs.
Building the evaluation set
An ai agent evaluation framework is only as good as its cases. Pull them from the real workflow, not from imagination. During week 0 to 1 of a pilot we measure the current path (median and slowest-case time per case, rework rate, queue depth) and collect sample data. That same sample becomes the first eval set, and the slowest and weirdest cases go in on purpose.
Group your inputs first. Several photos of one piece became duplicate products in one run, so we now cluster inputs before the agent sees them. A duplicate-input case belongs in the set permanently.
Then verify the checks themselves. A schema check of ours once flagged 1,153 pages when the real number was 95. An unverified checker is a false alarm generator, and teams that get burned stop trusting the suite. Run each new check against a known-bad and a known-good sample before it gates anything.
Speed counts too. For MEC we built an end-to-end test that checks about 28 rules per site in about 4 minutes, replacing a 40-minute manual check. When the test is that cheap, people run it on every change, which is the entire point.
How to plan the work this quarter
If you are working out how to test AI agents you are about to ship, take these steps in order. They mirror our typical pilot, which runs 4 to 6 weeks depending on how many systems the agent touches and how strict its checks need to be.
- Name one workflow, one owner and one metric that decides go or no-go. If you cannot name all three, stop and fix that first.
- List every action the agent can take and mark each reversible or not. Irreversible actions get a code gate or a human approval, no exceptions.
- Define the schema. Strict tool or JSON schema output, with real allowed values for categories, tags and statuses.
- Write the code gates and unit test them, including the test that proves nothing can reach a live state unreviewed.
- Build the eval set from real cases, add the weird ones, and verify each check against known samples.
- Deploy behind a feature flag to a small slice, then run shadow mode: the agent drafts, people approve, you tune on real traffic.
- Grant partial autonomy only where the gate is mechanical and failure is cheap. Publish before and after numbers, then scale, iterate or stop.
Two weeks of shadow mode costs less than months of quiet non-adoption. It is also where you find what the eval set missed, because real traffic is stranger than any sample.
Test the failure behavior, not just the happy path
Most of our painful incidents were not wrong answers. They were silent or operational failures, and each became a test.
- Silent success: a misconfigured job reported success in under a second with nothing produced. Our agents now fail loudly, and a test asserts that empty output is an error.
- Idempotency: a failed email still recorded its "sent" key, so it was never retried. Test that a failed side effect leaves no completion marker.
- Spend: three scheduled agents had no shared spend cap, and one paid endpoint was public when its secret was unset. Test the unset-secret case explicitly.
- Load: under a marketing spike, connection pools, uncapped functions and in-request email break first. We use a global concurrency cap and resumable steps under a 10-minute lease.
- Model outage: every agent needs a non-AI fallback. If the model is down, lead emails fall back to templates, and the output says which engine produced it. Pull the plug in staging and check.
One more habit: every manual fix during a live run becomes a code fix afterwards. If someone patched data by hand, there is a missing gate.
Where a delivery partner fits
You can build all of this in-house. The reason teams bring us in is usually the order of operations: they have a working prototype and no gates, no baseline, and no shadow-mode plan. Our AI agent development service covers that gap, from scoping the first release to running it in production, with human review built into the design and a non-AI fallback behind every agent.
If you are earlier in the process and still deciding which workflow deserves an agent at all, our guide to AI agents for small business walks through how to choose one. The MEC Artworks product-page build is the closest example of the testing approach above: a hidden-draft output, code-owned facts, a rule-based end-to-end check, and a person who publishes.
On cost: we quote each engagement after a short call about the workflow, because the drivers are integrations, review flow, data access and how strict the checks need to be. Running costs per task are small and estimated during the pilot, once we can see real input sizes and model choice.
If this is on your roadmap, tell us what you are trying to ship on the contact page and we will map a concrete next step.
