How to test an AI agent before launch

How to test an AI agent before launch, in four stages

The whole routine fits on one screen, so you can hold it in your head before you start:

• Stage 1 — Dry run: point the agent at sample or already-resolved inputs, with nothing that can act connected.
• Stage 2 — Draft only: let it produce real work, but keep every output as a draft inside the dashboard.
• Stage 3 — Three-run pilot: repeat the same job three times on fresh inputs and watch the corrections shrink.
• Stage 4 — Break it: feed one bad or missing input and confirm it stops with a clear reason.
• Go / no-go: launch only the parts that passed; hold the rest.

Stage one is the dry run. Give the agent a small set of inputs that cannot hurt you—last month’s already-closed tickets, a handful of made-up example records, a test list you typed yourself—and connect only the tools it needs to read, leaving anything that can send, pay, or delete unconnected. You are checking whether the agent understands the job at all, on data where a mistake has no audience.

Why it matters: a live connection is the difference between a wrong answer you delete and a wrong email a customer opens. Sample data removes the audience entirely, which is exactly what you want the first time you watch the agent work end to end.

How to judge it: the agent uses the sample inputs you gave it and nothing else, and its output is the shape you expected. The failure to watch for is an acting tool you assumed was out of reach but was quietly still connected, so the agent touches a real inbox or a live record during a run you thought was safe. Confirm on the dashboard that only the read tools are connected, not merely unused. The trade-off: sample data is tidy and real data is messy, so a clean dry run is necessary but not sufficient—it proves the agent can do the job, not that it can handle the strange inputs still waiting in real life.

Keep the first real pass draft-only

Once the dry run looks right, move to real inputs—but change one thing at a time. Keep the agent in a draft-only posture: it reads real data and produces real output, yet every result lands as a draft in the dashboard for you to read, and nothing is sent, paid, or deleted. This is the read, draft, approve, act pattern applied to launch day, separating the thinking, which is safe, from the acting, which is not.

Why it matters: real inputs surface problems sample data never will—an oddly formatted request, a customer name with an accent, a question the agent has not seen. A draft catches those where they are cheap, because a draft you disagree with costs a glance to fix and nothing has left your control.

How to judge it: read a full batch of drafts as if you were about to send them, and count how many you would change. The failure case is the draft that reads beautifully and is quietly wrong—confident tone, clean grammar, one fact that points to a source that never said it. Polished is not the same as correct, so check the claims, not the prose. The trade-off: draft-only means you are still doing the reviewing by hand, which can feel like you have automated nothing. You have not automated the sending yet; you have automated the drafting, and you are buying the evidence that will let you automate more.

Run a three-run pilot before you trust it

One good batch of drafts can be luck. A three-run pilot for AI agent workflows is the smallest test that tells luck and reliability apart: run the same job three separate times, each on fresh inputs, and change only the instructions between runs—never the tools. You are looking for consistency across different inputs, which is the thing you are actually buying when you automate.

Why it matters: reliability is a pattern, not a single event. The signal is not that run one was good; it is that runs one, two, and three were good in a row, and that the fixes you made got smaller each time instead of repeating.

How to judge it: track your corrections per run and watch the trend. Four fixes, then one, then zero is a workflow settling down. The failure case is fixing the same mistake on run three that you fixed on run one—that means the instruction is still ambiguous and the agent is guessing differently each time, not learning. The trade-off: three runs cost you three sittings before you get any automation at all, and the pull to declare victory after one clean run is strong. Resist it: the run you skip is the one that would have shown you the failure you are about to schedule.

Test the exceptions, not just the happy path

Every stage so far has watched the agent succeed. Before launch, spend ten minutes making it fail. Feed it a broken input on purpose—a request with the order number missing, a source link that leads nowhere, an instruction that contradicts itself—and see what it does when it cannot simply answer. Real inputs go wrong eventually, and the first time that happens should not be after you have looked away.

Why it matters: the whole promise of launching is that the agent runs while you are not watching. That promise is only safe if the agent stops and says so when it hits something it cannot handle, instead of inventing its way past the gap.

How to judge it: the agent should flag the problem in plain terms—“no order number found, cannot proceed”—and hold, not guess. The failure case is the dangerous pass: the agent fills the missing order number with a plausible one and hands you a confident, wrong answer that looks exactly like a correct one. That behaviour is invisible on a good day and expensive on a bad one, so hunt for it deliberately. The trade-off: engineering a failure feels wasteful when everything has gone smoothly, but a workflow you will trust unattended has to prove it fails loudly before you stop watching it.

Make the go/no-go call

Now you decide, and the decision is not all-or-nothing. Write down the handful of checks the workflow had to pass—used only the inputs you gave it, drafts were accurate, three runs held steady, it failed loudly on a bad input—and mark each one honestly. The point of naming the checks before launch is that you judge the agent against a standard you set while calm, not against your excitement on the day.

Why it matters: a launch is rarely a single switch. Reading and drafting can be safe long before sending and paying are, so a partial pass usually means launch the safe parts and keep the risky ones gated—not block everything or wave everything through.

How to judge it: a check that passes on its own evidence is a go; a check you are tempted to grade generously is a no-go, because “mostly fine” is how a shaky workflow talks its way into permissions it has not earned. The failure case is rewarding a good-enough pilot with the power to send, pay, or delete because momentum feels good—write access is the single hardest thing to walk back, so it should be the last thing you grant. The trade-off: launching only the safe parts today means the workflow is not fully automated yet, which is the right price. When the checks pass, the pre-launch routine hands off to the first-week rollout plan, where the schedule and the acting tools come on in stages rather than all at once.