AI Agent Observability: What to Trace, Measure and Review

An AI agent can fail without throwing a single error. The API answers, the workflow turns green, and the customer still gets the wrong answer. That is why AI agent observability is a different job from keeping a server up.

I've shipped 35+ agents into real businesses. The ones I trust have one thing in common: for any run, on any day, I can open a record and see exactly what happened. This article is the setup I'd use to get there.

What is AI agent observability?

AI agent observability is the ability to reconstruct any single run of an agent: what came in, which steps it took, which tools it called with which arguments, what it decided, what went out and what it cost. If you can answer "what exactly happened on run 4,812?" three weeks later, you have it.

That is more than logging errors. A normal app follows code you wrote, so when it breaks you read the stack trace. An agent chooses its own path. The same input can lead to a different sequence of tool calls tomorrow. So you need to record the path itself, not just the outcome.

In practice it comes down to three layers:

1. Traces: a step-by-step record of every run.
2. Metrics: a few numbers that tell you whether the agent is healthy overall.
3. Review: people checking a sample of outputs for quality, because no metric catches everything.

Why isn't normal monitoring enough for AI agents?

Normal monitoring checks whether a system is up and fast, and an agent can be both while giving wrong answers. A hallucinated order status comes back as a clean response. Uptime is 100%. Nothing alerts.

The failures that hurt with agents are quiet ones:

• A tool call returns an empty result and the agent fills the gap with a guess.
• A prompt change makes replies longer and slightly off-tone, and nobody notices for two weeks.
• A loop retries the same step forty times and the model bill doubles.
• A task drops out of the flow and nobody gets told.

This matters because plenty of agent projects die once they meet real work. Gartner's forecast is that over 40% of agentic AI projects get canceled by the end of 2027. Its reasons: costs climb, the value stays fuzzy, and risks go unmanaged. A 2025 MIT NANDA study put it even more bluntly: roughly 95% of the enterprise generative AI pilots it looked at had no measurable effect on the P&L. You can't defend an agent's value, or fix its cost, if you can't see what it does.

I wrote about the wider pattern in the five patterns that get agents pulled from production. Missing records is one of the five causes. This article is the fix in detail.

What should you trace on every agent run?

Trace every step of a run under one run ID: the trigger, the input, each model call, each tool call with its arguments and result, the final output, and the cost. If a step happened and you can't see it later, it's a blind spot.

Here's the minimum I'd record:

Two rules make this useful instead of noisy. First, nest the steps: a run contains model calls and tool calls, and you should see them in order, like a timeline. Second, store tool results, not just tool names. "Called the CRM" tells you nothing when the bug is that the CRM returned last year's address.

One warning: traces contain customer data. Decide who can see them and how long you keep them before you switch tracing on, not after.

Which numbers should you watch on an AI agent dashboard?

Five numbers cover most of what matters: completion rate, handoff rate, error rate, cost per run and edit rate. Put them on one screen per agent and look at them weekly.

1. Completion rate: runs that finished, out of runs that started. A drop means work is getting stuck or lost.
2. Handoff rate: how often the agent passes a task to a person. A sudden rise means something upstream changed, like a new kind of request. A sudden fall can be worse: the agent may be answering things it shouldn't.
3. Error rate: failed tool calls, timeouts and invalid outputs. Break it down by step so you know which integration is fragile.
4. Cost per run: tokens and model spend divided by runs. Spikes usually mean a loop or a prompt that grew.
5. Edit rate: the share of outputs a person changed or rejected before they went out. This is the closest thing you have to a quality score.

Edit rate is the one I'd watch first. If people approve drafts unchanged nine times out of ten, the agent is close to earning more autonomy. If they rewrite half of them, no dashboard improvement elsewhere matters. I use the same number in my customer service agent guide to decide when an agent may answer on its own.

Set alerts on two things only: completion rate dropping and cost per run spiking. More alerts than that and people learn to ignore them.

How do you check whether an agent's answers are actually good?

You check quality by having a person review a random sample of outputs on a fixed schedule, against a short checklist. Metrics tell you something changed. Review tells you whether it's wrong.

A simple routine that works:

1. Every week, pull 20 random runs per agent from the traces.
2. Score each against three or four questions. Is the fact correct? Does it follow the process? Is the tone right? Would you have sent it?
3. Log every failure with a one-line reason and the run ID.
4. Group the reasons. The biggest group is next week's fix.
5. Add each fixed failure to a small test set, so you can rerun it after every prompt or model change.

That last step is where observability turns into evaluation. Over time your test set becomes the list of things this agent must never get wrong again. When you change the prompt or switch models, you run the set first.

At high volume you can add automated checks, where a second model grades outputs. They're useful as a filter, but I wouldn't trust them until their scores match a person's on a few dozen runs.

Which AI agent observability tools should you use?

Start with whatever you already own, and switch to a dedicated tool when you have several agents or real volume. The tool matters less than recording the right fields.

If you can, send traces in OpenTelemetry format. It has semantic conventions for generative AI, and most tools above can read it, so you're not locked in if you switch later.

If you build with Claude Code, hooks can write a line to a log on every tool call. That gives you a basic trace without extra tooling.

What's the smallest observability setup worth building?

The smallest useful setup is one log row per run with input, steps, output, human outcome and cost, plus a weekly review of 20 runs. You can build that in an afternoon for a single agent.

Here's the order I'd do it in:

1. Give every run an ID and write one row per run.
2. Add each tool call with its result to that row or a linked table.
3. Record what a person did with the output: approved, edited or rejected.
4. Put completion rate, edit rate and cost per run on one view.
5. Block 30 minutes a week to review 20 random runs.

Everything else, from nested trace viewers to automated grading, is an upgrade on that base. Skip the base and the upgrades don't help, because you're still guessing.

If you're about to build your first one, my seven steps from process to production put this logging in from day one rather than bolting it on after the first incident.