Evaluation, Safety, and Observability

An agent that looks impressive in a handful of manual tests can still fail unpredictably in production, because manual testing only samples a tiny fraction of the situations an agent will actually encounter. Evaluation is the discipline of systematically and repeatedly testing agent behavior against representative scenarios, so that reliability is measured rather than assumed. Without it, every change to a prompt, a tool, or a model version is a gamble made on vibes rather than evidence.

The most useful mental model for agent evals is the same one used for software regression tests: a fixed set of representative scenarios with known-good (or known-acceptable) outcomes, run automatically whenever something in the system changes — a new model version, an updated prompt, a modified tool. This catches behavioral regressions before they reach production, the same way a unit test suite catches code regressions, and it turns "does this change make the agent better or worse" into a measurable question instead of a guess.

When an agent does something wrong, the only way to understand why is to be able to reconstruct exactly what it saw, what it decided, and what it called, at every step. Observability tooling that traces a full agent run — every tool call, every piece of retrieved context, every intermediate reasoning step — turns debugging from guesswork into inspection. Without tracing, a misbehaving agent is effectively a black box, and fixes become speculative rather than targeted at the actual point of failure.

Guardrails are the explicit constraints that prevent an agent from taking actions outside its intended scope, regardless of what the model's own reasoning concluded — limits on what data it can access, what destructive actions require explicit confirmation, what categories of output are blocked outright. The important distinction from prompt-level instructions is that guardrails should not depend solely on the model choosing to follow them; where consequences are serious, enforcement should happen at the system level, outside the model's own discretion.

Recent research has found that agents can produce harmful output as an incidental side effect of completing entirely normal, non-adversarial professional tasks — without any attempt to jailbreak or manipulate the model. This matters because it means safety evaluation can't only test for resistance to deliberately malicious prompts; it also needs to test ordinary, well-intentioned task completion for unintended harmful side effects, since the failure mode doesn't require an attacker to occur.

Beyond a binary pass/fail on task completion, mature agent evaluation typically tracks: task success rate against a representative scenario set, the rate of unnecessary or incorrect tool calls, context efficiency (is the agent achieving the same outcome with materially less wasted context), failure recovery (does the agent notice and correct its own mistakes, or does an early error silently compound), and safety-specific scenarios designed to surface harmful behavior even in benign-seeming task framings.

Anthropic's eval framework defines precise terms. A task is one test case with inputs and success criteria. A trial is one attempt (run multiple for variance). A grader scores some aspect; a task may have several graders with multiple assertions. The transcript (trace, trajectory) is the full record — every tool call, message, and intermediate result. The outcome is final world state — not the agent's closing message. A flight agent saying "booked" is transcript; a row in the bookings table is outcome.

Confusing transcript success with outcome success is the most common eval false positive.

Code-based graders — string match, regex, unit tests, static analysis, tool-call verification, token budgets. Fast, cheap, objective; brittle to valid variation.

Model-based graders — rubric scoring, natural-language assertions, pairwise comparison. Flexible for open-ended tasks; require calibration against human judgment.

Human graders — gold standard for subjective quality; use spot-checks to calibrate model graders at scale.

Production suites combine all three: code graders for invariants, model graders for nuance, human review for drift detection on judges.

Capability evals ask "what can this agent do?" — target hard tasks, expect low initial pass rate, hill-climb over weeks. Regression evals ask "did we break what worked?" — broad coverage of known-good scenarios, expect ~100% pass rate, run on every prompt/model/tool change.

Ship both. Teams that only run capability evals miss regressions; teams that only run regression evals stop improving ceiling performance.

Grade the path, not just the destination. Assertions should cover: required tools called in sensible order, forbidden tools never called, retrieved context relevant to the claim made, and outcome state matching expectations. Multi-turn agents can pass a final-answer string match while taking a dangerous or wasteful path — trajectory graders catch this.

Tools like Braintrust and LangSmith implement the same architecture: capture full traces, attach eval scores per span, run guardrails at tool boundaries, aggregate dashboards for pass-rate trends. Minimum viable observability for any agent harness: structured JSON logs per turn, trace ID across the session, tool parameters and results (redacted), latency and token metrics, and outcome verification results.

Case study: Pass rate on text-match evals was 96%, but production complaints spiked. Trajectory review showed the agent calling delete_workflow on debugging tasks. Fix: add trajectory assertion must_not_call_tools: [delete_workflow] unless task tag includes destructive-approved; add outcome check on workflow count. Regression suite caught the next model upgrade that reintroduced the behavior.

You can't improve what you don't measure. Evals for agents is an n-dimensional problem, as a failure can be a reasoning error, a tool description error, or an infinite loop in the harness.

• Demystifying evals for AI agents (Anthropic) — The best methodological guide on the difference between evaluating the model vs. evaluating the harness working together with the model, using the concept of trajectories (traces) instead of simple input/output validation.
• Braintrust and LangSmith — Documentation of the leading tools in observability. Studying their architecture teaches in practice how the tracing of non-deterministic loops and the creation of systemic Guardrails work.