The problem
Multi-step agents plan, retrieve information, call tools and revise their work. A single exact-match score can hide serious failures: a correct answer may come from unsupported evidence, while a safe escalation may be more valuable than a confident guess.
A practical evaluation structure
- Outcome: Did the system complete the user’s real task?
- Process: Were tool choices, intermediate steps and sources appropriate?
- Boundaries: Did it respect permissions, privacy and operational limits?
- Operations: Were latency, cost and failure recovery acceptable?
- Human hand-off: Did uncertainty trigger a useful escalation with enough context?
Key takeaway
Evaluate the decision process as well as the answer. Build a small, representative test set from real work, review traces, and treat human escalation as a designed outcome—not an exception.
What to implement first
Start with a narrow set of high-value tasks and define success in the language of the people who perform them. Record each agent trace, label failure modes consistently, and review the harness whenever tools, prompts, models or source data change.