The benchmark that asks
how, not just what.
Most agent benchmarks grade whether the agent produced the right final answer. UniClawBench grades whether the agent did the right things along the way, inside a live execution environment, across five distinct capabilities. The result is a score that separates what the model knows from what the framework enables.
First surfaced in Tandemly Briefing — 2026-07-13.
Grading the answer
hides the agent.
Most benchmarks measure whether an agent arrived at the correct final output. That design answers one question while leaving two others completely dark: where exactly did the agent go wrong, and was the failure caused by the model or the framework wrapping it?
The typical agent benchmark gives an agent a task in a simulated environment, waits for the agent to produce a final answer, and checks whether the answer matches a pre-recorded target. This is straightforward to implement and easy to compare across papers. It is also systematically misleading in two ways.
The first problem is opacity. An agent that got the right answer through poor reasoning looks identical to one that did everything correctly. An agent that stopped one step short of completion looks identical to one that stopped ten steps short. The score is a single bit of signal about a process that may have contained dozens of decisions.
The second problem is confounding. Every benchmark bundles a task, an execution environment, and an evaluation harness together. When the framework choices baked into the benchmark happen to match the style of one model and not another, the score reflects the match as much as the capability. Teams comparing models often end up comparing setups, without realizing it.
Researchers at HKU MMLab designed UniClawBench to address both problems at once: replace pre-recorded final answers with live checkpoint grading, and decompose the score into five named capability dimensions so model effects and framework effects can be separated.
If you run the same base model under multiple agent frameworks on the same task set, how much does the score change? And if you break that score into capability dimensions, can you tell whether the gap is the model's problem or the framework's problem?
400 tasks, live containers,
five dimensions.
The benchmark runs agents in actual Docker execution environments, not simulated sandboxes. A hidden supervisor agent monitors progress and issues feedback at each checkpoint without revealing the next checkpoint's criteria.
UniClawBench is built on three design decisions that each respond to a known failure mode in existing benchmarks. Together they produce a benchmark that is harder to game and more diagnostic about what is actually going wrong when an agent underperforms.
First, live Docker containers. Rather than simulating system state or pre-recording expected environment responses, each task runs inside an actual execution environment. The agent interacts with real software, real files, and real interfaces. There is no gap between how the benchmark simulates a tool and how that tool behaves in production.
Second, checkpoint grading via a hidden supervisor. Tasks are decomposed into discrete steps, each with an explicit success criterion. A supervisor agent monitors the agent's actions and provides multi-turn feedback at each checkpoint. Crucially, the supervisor withholds the criteria for the next checkpoint: the agent cannot reverse-engineer what it needs to do by reading the grader's signal. This prevents a common gaming pattern in which a capable model passes a benchmark by correctly inferring the grader's expectations rather than genuinely completing the task.
Third, five separated capability dimensions. Rather than producing a single composite score, the benchmark tracks skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination independently. A model that is strong on skill usage can be weak on exploration. An agent that handles long contexts well can struggle when tasks cross platforms. These patterns are invisible in a composite score.
The 400 tasks are published in both English and Chinese. Language coverage is frequently absent from composite benchmark scores: a model that performs well on English tasks may degrade on Chinese-language variants, and the drop is invisible when scores are averaged. Separate per-language scoring makes this visible.
The framework is
doing more work than you think.
The central finding is not about any one model's score. It is about the gap between a model's score under one framework and its score under another, running the same tasks.
When the benchmark team ran the same base models under different agent frameworks on the same task set, per-capability scores diverged meaningfully. A model that looked weak on one framework looked stronger on another. The divergence was not uniform: frameworks that handled long-context tasks differently produced different long-context scores for the same model, while skill-usage scores were more stable. This means capability assessments are framework-relative, not model-absolute.
Models strong on skill usage were not reliably strong on exploration. Long-context reasoning performance dropped in ways that were largely independent of other capability scores. Cross-platform coordination emerged as the most consistently difficult dimension across all models and frameworks tested. These patterns would be invisible in a composite score, because strong performance on easy dimensions masks weakness on hard ones.
Because the benchmark grades progress at each step rather than only at the final answer, it identified agents that stalled mid-task as a distinct failure pattern. Some agents that scored zero on final-answer matching had in fact completed the first several checkpoints correctly. That partial progress is real signal: it tells you the agent can start the task but cannot sustain it, which is a different diagnosis from an agent that fails immediately.
Note: this synthesis is based on the abstract, the originating briefing, and publicly available descriptions. The full paper methodology may contain additional nuance on specific models tested, exact score magnitudes, and statistical validation of the framework-effect finding. Treat the directional findings as well-supported but consult the full paper for precise numbers before citing them in technical work.
What to change
before your next eval.
The most direct implication is methodological. If your team evaluates agents by running one model on one framework and scoring final answers, you are measuring a combination of model and framework that you cannot yet disentangle. This paper gives you the design choices to fix that.
Where to go
from here.
Steps for putting this into practice.