Your workflows mix languages.
Your benchmarks don't.
Most agent benchmarks assume the world speaks one language. When researchers at Beijing Jiaotong University and Tencent's Weixin AI team broke that assumption across 67 real workplace tasks, state-of-the-art agents fell apart in a specific and predictable way: errors made early in the workflow didn't stay local. They cascaded.
First surfaced in Tandemly Briefing — 2026-07-18.
Two research traditions
that never spoke.
The field has excellent multilingual benchmarks. It also has excellent agent benchmarks. Neither one tests what happens when language variation hits a running workflow.
Multilingual benchmarks like XGLUE and M-MMLU have been refining their craft for years. They test whether a model can understand, reason about, and generate text across dozens of languages. But the format is almost always static: give the model a sentence or a document, ask a question, score the answer. There is no sequence of steps. There is no tool use. The model acts once and the evaluation is complete.
Agent benchmarks like SWE-bench and WebArena have a different focus. They test whether an agent can complete a long, tool-heavy task, often requiring dozens of steps, sub-goals, and tool calls. But the assumption embedded in most of them is that the whole workflow happens in one language. The instruction is in English. The tool outputs are in English. The expected answer is in English.
Real workplaces don't work this way. A commerce team handles orders from multiple markets. A legal team reviews documents that arrive in the client's language and must be summarized in the firm's working language. A localization workflow might receive a source document in French, apply translation tools, verify quality using a Japanese-language style guide, and produce output in Korean. Language switching is not an edge case in these workflows. It is the workflow.
What happens when an LLM agent must manage language variation not as a single translation step but as a property of the entire workflow? Do existing models hold up, and if not, where exactly do they break?
Sixty-seven workflows,
ten languages, five domains.
PolyWorkBench constructs real workplace workflows where language is woven through every step, not isolated to a single translation action.
The benchmark includes 67 tasks drawn from five workplace domains: commerce, knowledge work, legal analysis, localization, and manufacturing. Each task is a multi-step workflow requiring the agent to process inputs, invoke tools, reason across steps, and produce a structured output. Tasks are split into 29 baseline tasks (6 to 8 steps each) and 38 stress tasks (8 to 12 steps each), with an average of 8.5 steps per task.
Ten languages appear across the benchmark: English, Mandarin, Japanese, Korean, Vietnamese, Russian, French, Spanish, German, and Arabic. The key design choice is that language variation is placed at different points in each workflow. An instruction might arrive in German. The source document might be in Japanese. The required output format might specify French. In 88% of tasks, the agent must work in three or more languages within a single workflow execution.
No single grading approach is sufficient for multilingual long-horizon tasks. Executable verification is precise but blind to meaning. Structural rubrics reward progress but can't assess whether the French output is actually correct French. The LLM judge catches semantic problems but, as the paper's results show, has reliability limits of its own. The three together cover more of the failure surface than any one alone.
The harness moves scores
as much as the model.
The headline numbers are high. The story underneath them is about variance that doesn't come from the model at all.
The same model, switched between agent harnesses, swings 0.08 to 0.21 Pass@1 points on PolyWorkBench tasks. DeepSeek-v4-Flash reaches 0.796 inside the ClaudeCode harness. The same model in a different harness can sit measurably lower. GPT-5.5 paired with OpenClaw reaches 0.776. These are not small differences. They represent the harness layer accounting for as much outcome variance as model capability does on many tasks.
The benchmark is explicit about this: "Without disclosing the harness, benchmark scores lack meaning." A reported Pass@1 number attached to a model name, without a harness name, is incomplete information.
The paper's clearest finding is the compounding effect. A linguistic misunderstanding at step 2 does not stay local. It changes what the agent passes to step 3, which changes step 4's input, and so on through the workflow. By the end of a 10-step task, a single language-switch error can propagate into every downstream output.
Agents that handle isolated translation tasks well still degrade on PolyWorkBench, because the failure mode is a workflow property, not a single-step property. The error is not "this model cannot translate Japanese." It is "this model's translation error at step 3 corrupts the reasoning context for steps 4 through 10."
The semantic LLM judge, across all tasks, correlates with structural correctness at r=0.18. Within the regime where the judge expresses high confidence in its assessment, that correlation drops to r=-0.04. A more certain judge is no more accurate than an uncertain one. The paper recommends treating the LLM judge as a supplement for failure modes that deterministic checks miss, not as the primary scoring mechanism.
67 tasks across five domains is enough to surface systematic patterns, but the domain coverage reflects the team's available expertise and may not generalize to all multilingual workflow types. The benchmark is static: task difficulty is fixed at construction time rather than adjusted dynamically. The LLM judge's unreliability limits what the paper can conclude about semantic quality specifically, as opposed to structural correctness.
What to change
before the workflow ships.
The findings here are actionable for teams building agents that will touch multilingual data, whether they know it or not.
Where to go
from here.
Concrete next steps if you want to act on this research.