TL;DR
- Sauron is an agentic, evidence-first verifier for long-horizon, rubric-graded tasks. It interrogates state, artifacts, and the full trace; every verdict comes with verified, source-located receipts.
- It beats the strongest competing agentic judge, Gandalf, on all eight backbones we tested. On the best-performing one, GPT-5.6 Sol, Sauron reaches 0.958 vs. 0.923 unmet-F1 at 23% lower cost, grades tasks 4.2× faster, and widens its lead to 6.7 F1 points on process criteria.
- It catches reward hacking: across 637 paired fault-injection bundles, Sauron detects 97% of injected faults while flagging 2% of clean controls (Δfault = 0.950), where trajectory-blind judges stay below 0.1.
- On the TRAIL benchmark, Sauron leads Gandalf and TRAIL’s own LLM judge on every metric, even from a cheaper backbone.
Figure 1
What each verifier can see
Select a verifier.
Why agentic verification
The frontier of AI has moved beyond chatbots that answer questions to agents that do work. Today’s agents execute long-running, multi-tool workflows across real environments — editing files, updating databases, and sending messages. Success is no longer captured by an exact match or a passing test suite. Many requirements are better expressed as rubrics: Did the agent compute the value rather than hardcode it? Use the required source? Call the tools in the right order? These tasks are inherently semi-verifiable: we can specify success, but no single deterministic check can establish it. Evaluating them requires examining the result, the process, and the evidence left behind.
The stakes rise as AI enters its own development loop. OpenAI’s GPT-5.6 Sol post-training its smaller sibling Luna offers a concrete glimpse of that frontier. Such loops depend on reliably finding failure modes and separating real gains from apparent ones. The generation–verification gap, where models can recognize good work more reliably than producing it, provides the leverage. But once verdicts become training signals, every blind spot becomes an opening for reward hacking. Self-improvement therefore needs verifiers that go beyond the answer: they must inspect the state, retrace the process, and show their receipts.
Meet Sauron
Sauron is an agentic, evidence-first verifier: for each criterion, it adaptively interrogates the state, artifacts, and full agent trace, grounding every verdict in source-located, verified evidence (Figure 1). Three principles shape how it does this:
- State and trace, cross-examined. The governing invariant is simple: state is the source of truth; the trace is a source of leads. For outcome claims, the environment is authoritative, and no agent claim overrides a contradiction there. But some criteria cannot be resolved from final state alone, such as whether the agent used the required tools in sequence or computed a value instead of hardcoding it. For these process claims, the full trajectory becomes admissible evidence: a record of what happened and a map of where to probe next.
- Bounded, adaptive interrogation. Inspired by Recursive Language Models, Sauron keeps the full agent trajectory outside the model’s context and accesses it through bounded queries. It starts with lightweight evidence: the final response, targeted state probes, and previously verified findings. It inspects relevant trace regions only when the criterion demands it. This selective navigation ties context use to what a criterion demands rather than to how long the trace is.
- Evidence-gated verdicts, with receipts. Sauron’s unit of proof is an evidence atom: a claim paired with a closed-vocabulary evidence type and a machine-resolvable source locator. Admission is earned, not assumed. The locator must resolve, deterministic claims are checked mechanically, and semantic claims undergo an isolated faithfulness check: a model call that sees only the claim and exact source, not the judge’s reasoning. It asks one question: Does this source support this claim? Only admitted atoms can be cited. No verified atom, no award; undecidable defaults to unmet. This evidence trail makes every decision re-auditable offline (Figure 2).
Figure 2
Evidence is verified before it becomes citable
Sauron resolves every proposed atom against its source, checks exact claims mechanically when possible, and sends semantic claims to an isolated faithfulness check. Only admitted atoms can support a verdict.
The admission pipeline
Worked example — the atom flows through
To keep this practical for multi-criterion tasks, Sauron groups related criteria around a shared pool of admitted evidence. Each gets its own verdict and receipts, while investigative work is reused.
Results: better, cheaper, faster
To evaluate verifiers on work agents perform today, we drew a held-out, stratified sample of 100 tasks from an internal corpus of roughly 700 personal-assistant episodes. In each, an agent navigates a simulated personal-data environment to fulfill an open-ended request, such as audit my subscriptions or prepare a morning brief. The median run spans 53 trace events and 20 tool calls, producing roughly 3 artifacts, including dashboards, CSVs, and PDFs. Each task pairs a fixed Opus 4.8 trajectory with a human-labeled rubric averaging 26 criteria, for 2,586 in total. We evaluate eight verifier backbones, matched across verifiers in every comparison.
Figure 3 plots unmet-F1 against cost for every verifier and backbone. We report F1 on the unmet class because failures make up only 29.7% of the gold labels, and missing one creates the false reward verification is meant to prevent. Across all eight backbones, Sauron posts the highest F1 and costs less than Gandalf, the strongest competing agent-as-a-judge; above AutoRubric’s quality range, every point on the Pareto frontier belongs to Sauron.
Figure 3
Quality vs. cost across eight backbones
Unmet-F1 against total judging cost (log scale) — 100 tasks · 2,586 criteria · marker = verifier, color = backbone. Outlined circles are Sauron; the dashed line traces the Pareto frontier. The y-axis is broken to fit the text-only judge and Agent-as-a-Judge. Hover a verifier or a backbone to isolate it.
GPT-5.6 Sol is the strongest backbone in the sweep, and the rest of this post reports on it. There, Sauron reaches 0.958 unmet-F1 and 0.940 MCC to Gandalf’s 0.923 and 0.890, at 23% lower cost ($298 vs. $388; Table 1).
| Verifier | Unmet-F1 ↑ | MCC ↑ | Cost ↓ |
|---|---|---|---|
| Sauron | 0.958 | 0.940 | $298 |
| Gandalf | 0.923 | 0.890 | $388 |
| Archipelago | 0.919 | 0.885 | $2,058 |
| AutoRubric | 0.649 | 0.479 | $76 |
| Agent-as-a-Judge | 0.521 | 0.248 | $235 |
Table 1
Verifier quality on the benchmark at GPT-5.6 Sol
All 2,586 criteria; cost is per full run. Bold = best per column.
Sauron is also faster than Gandalf. On a 10-task slice of the benchmark, with both verifiers at their shipped internal settings, Sauron grades the set in 17 minutes to Gandalf’s 71 with one task in flight, 4.2× faster. Adding concurrent workers narrows the gap but never closes it: at eight tasks in flight, Sauron still finishes 2.1× sooner (Figure 4).
Figure 4
Sauron stays faster at every concurrency level
Wall-clock (log scale) for a 10-task slice of the benchmark with 1–8 tasks in flight, both verifiers at their shipped internal settings — the multiplier marks Sauron’s lead at each level.
Where the separation lives: process criteria
Process criteria assess how an agent produced its results. They ask whether required tools and sources were used, steps were taken in the right order, reported values matched tool outputs, and final claims agreed with the record. Final artifacts cannot settle these questions; the event history can. On the 411 process criteria within this benchmark, Sauron reached 0.968 unmet-F1, ahead of Gandalf at 0.901, Archipelago at 0.883, Agent-as-a-Judge at 0.821, and AutoRubric at 0.771 (Figure 5). Its lead over Gandalf is 6.7 points on process criteria against 2.4 on outcome criteria, and the process margin exceeds the outcome margin on every backbone.
Figure 5
Process criteria
F1 on the 411 process criteria — tool order, source access, computed-vs-hardcoded, claim-vs-record.
Catching reward hacking red-handed
To study how well verifiers distinguish genuine failures from plausible-looking success, we built a fault-injection suite of 637 bundles (13 fault families × 49). We injected failures from 13 realistic fault families, grounded in an interaction-centric taxonomy of agent failures, into real runs and paired them with clean controls to separate genuine detection from blanket rejection. Faults range from specification gaming and evaluation awareness to fabricated claims, trace–state mismatches, and environment-caused failures.
We score each verifier by Δfault = detection rate − false-alarm rate, so blanket rejection earns nothing. Sauron reaches 0.950, detecting 97% of injected faults while flagging 2% of clean controls (Figure 6). At their default settings, Gandalf and AutoRubric lack trajectory access and collapse toward blanket rejection, at 0.091 and −0.166. Passing both the full trajectory recovers much of the gap (0.675 and 0.716) but not all of it, and trace access alone does not resolve conflicts between what the agent said and what the environment shows: on trace–state mismatches, Sauron records +0.898, versus −0.265 for Gandalftrace and −0.939 for AutoRubrictrace.
Figure 6
Access to evidence separates verification from blanket rejection
Δfault (detection − false alarm) on the fault-injection suite, grouped by what each configuration can see — hollow marks are the trajectory-blind defaults.
Here, Sauron outperforms Gandalftrace at 4.6× lower cost and 2.2× lower latency per bundle, and it costs less per bundle than even the text-only AutoRubrictrace ($0.22 vs. $0.26). When verdicts become training signals, errors cut both ways. False approval rewards exploitation, while false blame incentivizes agents to conceal failures they did not cause.
In search of real failures on TRAIL
To further test Sauron on real agent failures, we evaluate it on TRAIL, a public suite of 148 human-annotated agent traces from GAIA and SWE-bench Lite, with 841 labeled errors across 21 categories. TRAIL poses the question a verifier exists to answer: not whether the agent finished, but where along a long trajectory it went wrong, and how.
Sauron leads Gandalf and TRAIL’s own LLM judge on every metric of both splits with both backbones we ran (Table 2). With GPT-5.6 Sol, its margins over the strongest comparator are 0.07 category F1, 0.18 location accuracy, and 0.19 joint accuracy on GAIA. Even on the smaller GPT-5.6 Terra, Sauron outperforms both Sol-backed baselines overall, which says the lead comes from how the trace is searched, not from which model searches it.
| Verifier | GAIA | SWE-bench Lite | |||||
|---|---|---|---|---|---|---|---|
| wF1 | Loc | Joint | wF1 | Loc | Joint | ||
| GPT-5.6 Sol | |||||||
| Sauron | 0.533 | 0.725 | 0.444 | 0.676 | 0.434 | 0.098 | |
| Gandalf | 0.463 | 0.527 | 0.250 | 0.610 | 0.244 | 0.062 | |
| JudgeTRAIL | 0.461 | 0.547 | 0.215 | 0.469 | 0.209 | 0.040 | |
| GPT-5.6 Terra | |||||||
| Sauron | 0.514 | 0.768 | 0.398 | 0.675 | 0.572 | 0.064 | |
| Gandalf | 0.366 | 0.448 | 0.149 | 0.543 | 0.200 | 0.057 | |
| JudgeTRAIL | 0.376 | 0.484 | 0.162 | 0.463 | 0.169 | 0.047 | |
Table 2
TRAIL: error categorization and localization in public agent traces
148 traces (117 GAIA, 31 SWE-bench Lite), 21 error categories. wF1 = support-weighted category F1; Loc = location accuracy; Joint = joint (span, category) accuracy. JudgeTRAIL is the benchmark’s official single-pass judge. Bold = best per backbone.
Conclusion
We introduced Sauron, an evidence-first agentic verifier that outperforms other agentic judges while costing less, running faster, and resisting reward hacking. The paper covers the architecture and the ablations behind each mechanism, and Sauron is available on GitHub. Next, we are taking Sauron into the training loop, where reliable reward-time verification matters most.