Agent leaderboards mislead — benchmark against your own tools

Public agent leaderboards looked authoritative until teams actually shipped with the model that ranked first. Then the tool-call failure rate in production bore no resemblance to the score on the chart. The reason is structural: every published benchmark tests a fixed, curated set of generic tools, and your internal stack is almost certainly none of those things. The honest position is that model choice for agentic workloads is unanswerable without an eval harness pointed at your own tool surfaces.

What changed in how the field measures agent capability

The Hugging Face team's post "Is it agentic enough?" makes the structural critique explicit. The dominant agentic benchmarks — including heavily cited suites built around web search, calculators, and file I/O — share a single design constraint: tool surfaces are fixed and standardized so results are comparable across labs. That is the right property for a comparative leaderboard. It is the wrong property for an operator trying to pick a model for an internal agent that calls a proprietary ERP, a custom document-retrieval API, and a legacy SOAP service wrapped in a thin REST shim.

The post also surfaces the multi-step compounding problem. A model that achieves 80 % single-step tool-call accuracy doesn't deliver 80 % task completion on a 5-step chain — it delivers roughly 33 % (0.8⁵). Leaderboards almost always report single-step or short-horizon scores. The gap between those scores and real-world task completion widens non-linearly as chain length grows, which is precisely where enterprise agent work lives: multi-hop lookups, conditional branches, write-back steps.

Open model releases accelerated this problem in 2025 and the first half of 2026. When a new model posts a 3-point improvement on a public benchmark, the model card travels fast. The tool-call schema it was trained on — whether it prefers strict JSON function-calling, a ReAct-style scratchpad, or a code-execution loop — gets far less attention in the announcement, but it determines whether the model degrades gracefully or catastrophically on your schema.

Why this changes the math on owning your agent evaluation

Most teams at the 50–500 FTE scale do not have eval infrastructure for agents. They have a staging environment, a handful of manual tests, and someone who runs the model against a few representative prompts before a deployment. That was adequate when LLMs were completing text. It is not adequate when the model is calling tools with side effects — writing records, triggering workflows, updating state.

The strategic implication is that the decision "which model should we use for our internal agent?" is currently being made without valid data at most companies. Teams are pattern-matching on benchmark headlines, vendor reputation, or cost-per-token. All three are reasonable tie-breakers when two models are otherwise equivalent for your task. None of them is a substitute for measuring tool-call accuracy, schema adherence, and multi-step completion rate on your actual tools.

There is also a sourcing risk that cuts the other way. If you have not built the eval harness, you also cannot detect regression when you update the model or change a tool's schema. Agent stacks are not static — tools evolve, models get swapped, prompt templates drift. Without continuous eval against representative traces, you are flying without instruments every time something changes.

The Monday-morning move

The first question is whether you have the preconditions to run a meaningful eval. Work through this sequence before spending time on model comparison:

• Do you have tool call traces from production or staging? If not, instrument your agent framework to log every tool call, the input schema sent, the raw model output, and the parsed result. 2 to 3 days of traffic at even moderate volume gives you enough to build a replay corpus.
• Do you have ground-truth labels for a subset of those traces? You need at minimum 50 to 100 labeled examples — correct tool selected, correct arguments, correct final answer — to get a signal that isn't noise. Domain experts on your team, not ML engineers, should label these.
• Is your tool count high enough that model choice matters? Below roughly 5 tools, almost any capable open model will perform similarly. Above 10 tools, especially with overlapping or ambiguous schemas, model choice starts to matter significantly. Above 20, it is probably the dominant variable in agent reliability.

Once those preconditions are met, the actual eval is not expensive to run. Pick 2 to 3 candidate models — at minimum the one you're currently using, one challenger from the current open-weight cohort, and a cost-optimized smaller model as a baseline. Run all three against your replay corpus with identical prompts. Measure single-step tool-call accuracy, argument schema validity, and end-to-end task completion on multi-step traces separately. The deltas will tell you more in an afternoon than a year of reading leaderboard posts.

If your team does not have the capacity to build the harness this week, the minimum viable version is a structured log and a labeled sample. You can build the replay runner later. Don't let perfect eval infrastructure block you from starting to collect the data you'll need.

What it costs, and what it saves

Building a minimal agent eval harness takes roughly 2 to 4 engineer-weeks to get right — instrumentation, a replay runner, a labeling workflow, and a reporting layer simple enough that a product manager can read it. That is real cost, and it will feel like infrastructure tax in a quarter where the roadmap is already full.

The alternative cost is harder to see but larger. Teams that skip eval typically discover model-specific failure modes in production. At best, that means a degraded user experience for a sprint cycle while you hot-fix prompt templates. At worst — when agents have write access to systems of record — it means data integrity issues that take weeks to untangle. The regression risk alone justifies the infrastructure investment: every model update without a harness is an uncontrolled change to a system with side effects. The companies that will have a durable advantage in agent reliability over the next 18 months are not the ones that picked the right model from a leaderboard. They are the ones that built the measurement layer first.