The failure starts
before the agent runs.
First surfaced in Tandemly Briefing — 2026-07-28.
A researcher at the University of Chicago built a seven-criterion scoring system for agent context quality and isolated it from behavioral metrics entirely. Then ran 300 controlled evaluations holding the model fixed and varying only the context. What changed between poor and structured context: critical failures dropped from 4.11 to 1.33 per session. The lesson is about where reliability actually comes from.
Measuring outputs
to diagnose inputs.
Most agent evaluation runs the agent, counts failures, and calls that a quality signal. There's a circularity problem buried in that approach that the field has mostly ignored.
When an AI agent misbehaves, the standard sequence is: run the agent, observe what it does, count the things that went wrong. If it hallucinated, if it broke a guardrail, if it ignored a constraint or spent three times the expected tokens on a simple task, those outcomes show up in a behavioral metric and get logged as failures.
But every agent decision is shaped by what's in its context. The system prompt that defines the agent's role. The tool definitions that specify what functions it can call and with what schemas. The retrieved knowledge grounding its answers. The guardrail instructions limiting what it's allowed to do. The phrasing choices that either invite or resist injection attempts. These are the inputs, and they vary enormously in quality across production deployments.
The circularity problem: if you evaluate context quality by running the agent and counting failures, the score depends on the same outputs it's supposed to predict. You can't use behavioral metrics to grade the context that produced the behavior without losing the ability to use context quality as a predictive signal.
There's also a timing problem. By the time behavioral failures surface in a testing pipeline, the context has already shipped. The failure arrived late. What's needed is a way to grade the context before the agent runs, as a preflight check rather than a post-mortem.
Can context quality be scored independently, before any behavioral tests run, in a way that actually predicts which failure modes to expect? And if so, how much does context quality explain about reliability variation across deployments?
Seven criteria,
scored before the run.
Bousetouane built ProofAgent-Harness, an open-source multi-juror infrastructure for adversarial agent evaluation, and used it to measure context quality across seven dimensions. The scores went down before the behavioral tests ran.
ProofAgent-Harness uses a consensus-based scoring approach: multiple independent jurors assess each criterion and the result is a consensus score, not a single reviewer's judgment. This reduces the subjectivity problem in qualitative context assessment.
The crucial design choice was sequencing. Context quality scores were recorded before behavioral evaluations began. Behavioral outcomes were only examined after scoring was complete. This sequencing breaks the circularity: the context score cannot be influenced by the behavioral outcomes it is predicting, and the evaluator scoring context cannot rationalize the score backward from observed failures.
The controlled study ran 300 multi-turn evaluation sessions across three deployment domains: customer support, healthcare claims processing, and legal drafting. The model was held fixed throughout. Only the context varied, across three levels. Poor context meant minimal system prompts, generic tool definitions, and no explicit guardrails. Structured context meant well-specified roles, typed tool schemas, and explicit constraint instructions. Hardened context added injection-resistant phrasing and explicit handling of adversarial inputs on top of structured.
Scoring context quality before behavioral tests run is not a procedural formality. It determines whether the score is genuinely predictive or just retrospective rationalization. A context score assigned after you've seen the failure rate is a description. A context score assigned before is a prediction. The paper's contribution rests on that distinction.
Criteria predict
their matched outcomes.
Three results across 300 sessions. The main one is that context quality is a real variable with a large effect on critical failure rate, independent of the model.
The grounding sufficiency score predicted hallucination resistance. The guardrail coverage score predicted manipulation resistance. Each of the seven criteria pointed to a corresponding class of failure mode. This is the non-circular validation the design was built for: the context scores went down before the behavioral tests ran, and the predictions held when the results came in.
This result is more useful than it might first appear. It means context quality can function as a diagnostic vocabulary. If you know a deployment's grounding sufficiency score is low, you know which tests to weight most heavily in behavioral evaluation. You don't have to run the full evaluation before you know where to look.
Moving from poorly-specified to properly-specified context cut critical failures by 68%, from 4.11 to 1.33 per session. This happened before any safety hardening was applied. The majority of the reliability gain came from basic structural work: clear role definitions, typed tool schemas, explicit constraint statements. The "hardened" level brought further improvement, but the largest single step was from poor to structured.
This matters for resource allocation. Teams often invest in adversarial red-teaming and injection-hardening before addressing basic context quality. The data suggest that's the wrong order.
Even at the hardened context level, critical failures did not reach zero. A well-specified, injection-hardened context produces fewer failures, not no failures. The seven-criterion score tells you what to expect; it doesn't eliminate the risks it names.
This is an honest limitation the paper does not hide. Context quality is a leading indicator of reliability, not a guarantee of it. Teams that treat a high preflight score as permission to skip behavioral testing will be surprised.
Three deployment domains (customer support, healthcare claims, legal drafting), one model family, one harness. The seven criteria and their matched behavioral predictions have not been validated across a wide range of model families or deployment types. The paper is a first demonstration, not a validated framework. The claim that grounding predicts hallucination, for example, is supported by these 300 sessions; it needs more domains to be general.
What to do with
context as a variable.
The practical shift this research asks for is not adding a new tool. It's adding a new step before a step you already take, and treating context quality as something measurable, not intuitive.
Where to go
from here.
If you want to go deeper on context as a measurable variable in agent reliability.