A Practical Guide, Part 6 of 15: Evaluation and Verification

Host: Right. A beautiful plan is still just a plan. Today is about checking what happened when the plan met the world.

Host: Picture this. You've built a customer support agent. You run it once on a refund request, it nails it, everyone high-fives, and you ship. A week later it's refunding the wrong orders. What went wrong?

Co-host: Honestly, I've been that engineer. You ran one demo and called it a test.

Host: So, vocabulary first. A task is one problem with a defined success condition. Each attempt at that task is a trial, and teams run several trials because model outputs vary from run to run.

Co-host: And the transcript is everything that happened during a trial: the messages, the tool calls, the tool results. The thing you'll be reading at two in the morning.

Host: Then the grader is whatever scores the trial, and a collection of tasks with their graders is your eval suite. The crucial word, though, is outcome. The outcome of a trial is the final state of the environment, not what the agent claims at the end of the transcript.

Co-host: That one bites people. The agent says, I've updated your booking, have a lovely day, and the database is untouched.

Host: Exactly. A confident summary is a claim. The database row is evidence. Tau-bench, which we'll come back to, builds on this: it grades by comparing the database at the end of the conversation with an annotated goal state.

Host: So, in practice: never trust a single run of a non-deterministic system, and grade the world, not the words.

Co-host: Okay, so who does the grading? Because I've seen teams argue about this for weeks.

Host: Three families. Code-based graders: string matches, unit tests, a query against the final state. They're fast, cheap and reproducible, but brittle. Return the right answer in a slightly different format and they fail you.

Co-host: I once had a grader reject a correct date because the agent wrote the month as a word. Technically a failure. Spiritually a pass.

Host: Then model-based graders, the language model as judge. Flexible, can handle open-ended answers and rubrics, but they need calibration. The MT-Bench paper looked at exactly where they go wrong: position bias, verbosity bias, self-enhancement bias, and limited reasoning.

Co-host: Unpack those for a practitioner. Position bias is preferring whichever answer is shown first?

Host: That's the idea. Verbosity bias is liking longer answers whether or not they're better, and self-enhancement is a judge favouring output that sounds like its own. The mitigation in general terms is swapping the order and averaging, and writing rubrics that reward correctness rather than length.

Co-host: But does it actually agree with people?

Host: The same paper's headline: strong judges like GPT-4 reached over 80 percent agreement with human preferences, roughly the level humans reach with each other.

Co-host: Steelman time. That sounds like humans are optional.

Host: The strongest version is: for chat-style preference comparisons, a good judge is about as consistent as a person. But that was measured on that kind of task. My view is that agreement on which reply reads better doesn't transfer automatically to whether a multi-step agent left the system in a correct state. You have to recalibrate on your own data.

Co-host: And humans are the third grader.

Host: The gold standard, and the slowest and most expensive. So the practical mix is code wherever there's a checkable state, a model judge for the fuzzy parts, and humans sampling regularly to keep the judge honest.

Host: Now the distinction that changes how you read every agent leaderboard. Suppose you run each task eight times.

Co-host: pass at k asks: did at least one of those k attempts succeed? That's great for coding with a test suite, because you can just keep the one that passed.

Host: And pass to the power k, sometimes written pass hat k, asks the opposite: did all k attempts succeed? As k grows, the first number climbs towards one, and the second slides towards zero, unless the agent is genuinely reliable.

Co-host: For a support agent, the second one is the business metric. Customers don't get to pick the best of eight.

Host: Tau-bench is the worked example. Even state-of-the-art function-calling agents like gpt-4o succeeded on under half the tasks, and in the retail domain scored below 25 percent when asked to succeed on all eight trials.

Co-host: So it's roughly a coin flip once, and terrible consistently. That gap is the whole story.

Host: And this idea is reaching industry. SecurityScorecard announced on October 29, 2026 what it calls the world's first AI benchmark for third-party risk management, testing models on risk and cyber tasks and how well they withstand adversarial manipulation.

Co-host: With a big asterisk: it's a vendor press release, so results are self-reported for now. They've open sourced the framework so people can test models and agents and submit to a public leaderboard.

Host: They also say it rotates task inputs, protects answer keys and adds safeguards against memorisation. Whether those hold up is for independent testers to show, but the direction is right.

Host: So report both numbers for every agent you build, and decide in advance which one your users actually experience.

Co-host: Let's talk public benchmarks. My manager sees a score and asks why we're not using that model.

Host: Start with SWE-bench: 2,294 software problems taken from real GitHub issues and pull requests across 12 popular Python repositories.

Co-host: Real issues is the strength. The weakness, as I see it, is it's Python-only and drawn from a fixed set of public repos, so it says little about your codebase.

Host: AgentBench takes breadth: eight distinct environments testing a model's reasoning and decision-making as an agent.

Co-host: Breadth is nice, but eight environments means each is fairly thin compared with one real deployment.

Host: And tau-bench adds simulated users and policy rules, graded on final database state. Which leads to my favourite failure story. Anthropic reports that Opus 4.5 solved a tau-two-bench flight-booking task by finding a loophole in the airline policy. It failed the eval as written while giving the user a better outcome.

Co-host: So the agent was too clever for the answer key. That's a grader bug, not a model bug.

Host: Or a policy bug. Either way, a benchmark is a frozen opinion about what correct means. Public scores tell you roughly where a model sits, never whether it fits your tasks.

Co-host: Practical bit. Monday morning, I have no evals. What do I do?

Host: The guidance in this series' plan is twenty to fifty tasks, drawn from real failures: the tickets, the bug reports, the transcripts where things went sideways. Not invented edge cases from a whiteboard.

Co-host: Small enough to read every result. I like that.

Host: Then grade outcomes, not paths. If you insist the agent calls tool A then tool B, you punish valid alternatives, like that loophole booking. Check the end state.

Host: Next, read the transcripts. A score says something failed. A transcript says the grader was wrong, or the tool returned garbage, or the agent gave up. You will find grader bugs, and that's the point.

Co-host: And saturation?

Host: When everything passes, the suite has stopped teaching you anything. Retire the easy tasks into a regression set and add harder ones from fresh failures. In practice an eval suite is a living thing, more garden than monument.

Co-host: Last piece. Evals happen offline. Can the agent check itself while it works?

Co-host: In coding agents I like Stop hooks: when the agent tries to finish, a hook runs the tests, and if they fail it's sent back to work.

Host: And adversarial review subagents: a second agent whose only job is to find what's wrong. The trade-offs are cost and latency, and two models can share the same blind spot, so it doesn't replace your offline suite.

Co-host: Plus a hook can be gamed if the agent can edit the tests. Lock those down.

Host: So: every claim the agent makes about finishing should trigger a check it can't talk its way past.

Host: The one takeaway: measure consistency, not a lucky run, and grade the final state of the world.

Host: This was part of our series Agentic AI: A Practical Guide. The show notes list every source, and this episode was made with AI voices. Follow AI Shadow for the rest of the series. Goodbye for now.