Faster. Cheaper.
Not good enough yet.
First surfaced in Tandemly Briefing — 2026-07-30.
Baidu researchers built a benchmark where every office task carries an economic price tag: human labor time and cost. Every model they tested completed tasks faster and cheaper than a human worker. None produced output at human deliverable quality. The gap between "cheaper than a human" and "replaces a human" is real, measurable, and still wide.
Pass/fail misses
the point.
Standard agent benchmarks ask whether a task was completed. That question made sense when agents were unreliable enough that completion itself was the goal. Agents are now reliable enough that "did it finish" is no longer the useful question.
Most agent benchmarks were designed when the dominant failure mode was an agent that could not complete a task at all. A model that gets stuck in a loop, generates nonsense, or simply gives up does not complete the task, and task completion rate captures that failure cleanly.
But frontier models now complete a large fraction of well-defined tasks on the first try. The leaderboards reflect this: completion rates are high, scores converge, and the gaps between models on standard benchmarks are often smaller than the gap between any model and the upper bound of what the benchmark rewards. The metric is still moving, but it is moving more slowly, and the improvements it measures are increasingly incremental.
The more pressing question for anyone deploying agents on professional work is not whether the task was completed, but whether the completed output is actually usable. A research summary that needs two hours of heavy editing before it can be sent to a client is not an equivalent substitute for one written by a skilled analyst. A formatted document with structural errors is not the same as a clean deliverable. Completing the task and producing a usable deliverable are not the same thing, and the difference between them has an economic cost that standard benchmarks do not capture.
What happens when you replace "did the agent complete the task" with "is the agent's output worth what it would cost a human to produce the same quality"? Does the economic case for agent deployment hold up under that framing?
Office tasks, priced
against human labor.
OmegaUse-OfficeVal is a benchmark of long-horizon office-suite tasks, each annotated with a human labor time estimate and a cost proxy. The benchmark enables two metrics standard evaluations omit: quality relative to human output, and quality per dollar of agent cost.
The tasks are the kind of work that shows up in professional office environments: documents that require multiple synthesis steps, formatted outputs with structured requirements, tasks where the quality of the final product is visible and assessable. "Long-horizon" means each task requires multiple intermediate decisions before the output is complete, not a single-step answer.
Each task carries two economic tags. The first is a human labor time estimate: how long a skilled worker would need to produce an acceptable deliverable for that task, based on professional time standards. The second is a derived cost figure drawn from labor rates for that kind of work. Together, they give a concrete human-cost baseline for each task in the benchmark.
With that baseline in hand, the benchmark computes two things for any agent run. Quality is measured by a rubric applied to the output deliverable, not just to task completion. Cost is measured as total agent spend, API calls included. Quality-per-dollar for the agent is then compared against the quality-per-dollar from a human worker on the same task, using the economic tag as the human data point.
Most cost-aware agent research measures how much the agent spends (API tokens, latency) relative to task success rate. OmegaUse-OfficeVal measures something different: how much value the agent produces relative to the human labor it would replace. The two questions are related but not the same. An agent that costs $0.08 per task where a human costs $40 per task looks extremely cheap. If the agent's output quality requires $30 in human correction, the agent costs $38 per usable deliverable, not $0.08.
Cheaper. Faster.
Still not there.
Every model tested completed tasks faster than a human worker and at lower direct agent cost. None produced human-quality deliverables on the long-horizon tasks. The economic case for full automation did not close.
Models complete the tasks. They do so faster than a human worker and at lower direct API cost. On those two dimensions, the benchmark results look like a success story for agent deployment.
The incomplete part is that "completing the task" and "producing a deliverable that does not require substantive revision" are not the same thing for long-horizon professional work. A task that is technically complete but requires an hour of human editing to become usable is not a time-and-cost win. The benchmark makes this distinction explicit by scoring the output quality separately from task completion.
None of the models tested produced outputs that matched human deliverable quality on the long-horizon office tasks. The gap was not marginal. Outputs required substantive review before they were ready to use as professional products.
This does not mean agents are bad at office work in general. The specific failure mode here is long-horizon tasks where quality is assessed on the deliverable structure and professional form, not just on factual content. Shorter, more structured tasks may tell a different story. But for the class of work where professional output quality is the standard, the economic case for full replacement, rather than assistance, does not yet hold.
The common framing of agent adoption as "faster and cheaper than hiring" relies on an implicit assumption: that the agent's output is usable as delivered. OmegaUse-OfficeVal shows that assumption fails for long-horizon office tasks at current capability levels. When the cost of human review and correction is added to the agent's direct cost, the economic advantage narrows or disappears, depending on the task and the required quality level.
The benchmark also surfaces a useful diagnostic: the tasks where agents score highest on speed and direct cost are not necessarily the tasks where they produce the strongest deliverables. The correlation between "fast and cheap" and "high quality" is not assumed in the benchmark design, and the results suggest it should not be assumed in deployment decisions either.
The benchmark reflects Baidu's definition of office-suite tasks and their labor cost estimates. Different organizations, task definitions, and quality rubrics would produce different numbers. The human labor time estimates are derived from professional time standards, not measured from a controlled comparison group. The results describe what happened at one point in model capability development and will change as models improve. The core contribution is the methodology: attaching economic tags to benchmark tasks so the comparison is possible, not a fixed claim about which models close the gap.
What this means
for how you evaluate.
The benchmark design is the contribution, not just the result. The question OmegaUse-OfficeVal asks, "is the agent worth what it would cost a human to do the same quality work," is a question any team can apply to their own evaluation setup with modest additional effort.
Where to go
from here.
The benchmark methodology is the most portable takeaway from this paper. Several of the steps below can be applied to an existing evaluation setup without waiting for a new benchmark release.