5 dashboard signals that tell you whether an agent is healthy

Start with outcomes, not activity

An agent can make hundreds of flawless tool calls and still hand you something unusable. Counting the calls tells you the machine ran; it says nothing about whether the machine helped. So begin every dashboard with one question: did the promised output arrive, on time, in the right place, good enough to use?

Five signals answer that question. Watch these, not the raw log:

• Useful completion — did the run actually produce the output someone needed?
• Review time — how long a person spends checking each result before trusting it.
• Correction rate — how often the result needs real edits, not just a glance.
• Intervention rate — how often the agent stalls and asks you to step in.
• Cost per useful result — total run cost divided by the outputs you actually kept.

How to judge it: if your dashboard can show a run as “successful” while a person got nothing they could use, it is measuring activity, not outcomes. A realistic failure case: a daily support summary reads 100% complete all month, yet three mornings it posted after standup—the badge tracked the system, not the reader. The trade-off: outcome signals cost a few minutes of human judgment to record, while system metrics are free; you are trading a little effort for the only numbers that tell you the work was worth doing.

What each signal is telling you

Each signal catches a different kind of trouble, so read them together—one number rarely tells the whole story, and a healthy agent keeps all five in a comfortable range.

• Useful completion. Not “the run finished” but “the run produced the result you promised.” A summary that posts on time yet skips the day’s biggest issue is a miss. When this drifts below your own bar, tighten the success rule before you touch anything else.
• Review time. The minutes a person spends checking each output. Rising review time is the quietest warning there is: the agent still “works,” but it is slowly handing back a second job. Track the median so one awful run doesn’t distort the picture.
• Correction rate. How often you rewrite the result instead of accepting it. A few edits are normal; the same edit every day means a missing rule or source, not a weak model.
• Intervention rate. How often the agent stops and asks for help. Some questions are healthy caution. A sudden spike usually means the task outgrew its instructions or a connection is flaking.
• Cost per useful result. Total run cost divided by the outputs you kept—not the runs you started. This is where retry loops hide: an agent that quietly tries five times to succeed once can look fine until you divide by what you actually used.

A realistic failure case: completion holds at 100%, but correction rate climbs all week. Someone reads it as a model problem and swaps to a pricier model—cost per useful result doubles and the corrections continue, because the real cause was an out-of-date price list the agent kept citing. The trade-off: five signals are more to watch than one green badge, but each number you choose to ignore is a failure mode you have agreed not to see.

Read the pattern, not one bad run

A single failure is usually noise. Three slower runs in a row, a correction rate that climbs all week, or the same clarification requested every morning is a signal. The unit of judgment is the trend, not the incident.

One habit keeps you honest: compare the last seven runs with the previous seven before you change anything. If the recent window is clearly worse, you have a regression worth investigating. If it looks the same, the run that annoyed you this morning was probably just variance.

A realistic failure case: one summary lands late, the owner immediately swaps the model, and the next day’s run is late for an unrelated reason—now two things have changed at once and neither can be trusted. The trade-off: a small window of seven runs reacts quickly but jumps at noise; a month-long window is stable but can hide a real regression for weeks. Match the window to how fast the work actually moves. How to judge it: if you can’t point to a trend across several runs, wait for one before you act.

Set alerts that lead to action

An alert is a promise that something needs a human right now. If nothing changes when it fires, it isn’t an alert—it’s a log entry with a notification attached. Alert only when a person can change the outcome: a run is stuck, a connection expired, a risky action is waiting for approval, or results have failed review several times in a row.

Write each alert so it can be answered from a phone: name the task, the problem, and the safest next step. “Daily support summary hasn’t posted—last attempt failed on an expired channel connection; reconnect or skip today” tells you what to do. “Run 4821 failed” just sends you hunting.

A realistic failure case: an alert fires on every failed run, including the harmless ones the agent retries and recovers from on its own. Within a week the owner swipes them all away by reflex—and misses the one Monday the summary genuinely never posts. The trade-off: quieter alerting risks missing a real problem, while noisier alerting guarantees you will eventually ignore the real one. When unsure, alert less and review more: a missed alert you catch in the weekly review costs a day, but alert fatigue costs you every alert after it.

A 10-minute weekly review

Health isn’t a live dashboard you stare at; it’s a short habit you keep. Once a week, in about ten minutes, walk the same five steps in order:

1. Open the failed and overdue runs first—the outcomes, not the error traces.
2. Read the two results with the longest review time; that is where hidden work lives.
3. Look for a correction you made more than once—it usually points to a missing rule or source.
4. Pause any workflow whose output nobody used this week.
5. Change one thing—one rule, one source, or one permission—and note the date so next week’s comparison means something.

A realistic failure case: during one review the owner tightens the success rule, swaps a data source, and widens a permission all at once; the next week improves, but there is no way to know which change helped—or whether the wider permission quietly added risk. The trade-off: a fixed ten-minute cadence won’t catch everything a run-by-run audit would, but it is a habit you will actually keep—and a review you skip protects nothing.