TANDEMLY
Understanding AI innovation in tandem with AI
Humans + AI,
Making Sense of AI
Tandemly is where we make sense of AI in public — synthesizing research, tracking the daily firehose of news, and distilling the practices that actually work. We also ship projects in tandem with AI: sometimes weird, sometimes useful, always a way of seeing what happens when humans and machines build together.
Projects
Cutthroat
Ruthless word game — steal letters and words from your opponents.
Gaming Red Pandas
Browser-based indie games — whimsical solo, local, and multiplayer panda adventures.
Research & Writing
More Reviewers, Same Single Point of Failure.
When multi-agent reviewers share the same upstream data source, their votes are structurally one. The Structural Epistemic Cut proves quorum size is not a safety dial, and shows "add another reviewer" provably fails when the new reviewer reads the same inputs. He & Yu, 2026.
Staleness belongs on the wire.
Agent caching across sessions fails silently when servers change. Invalidation contracts attach version stamps so clients evict stale entries before trying them. Compliance is a model property: 100% first-try on Haiku 4.5, under 11% on Sonnet 5. Wu & Canedo, 2026.
The prompt is not working memory.
Scroll replaces the growing text buffer with a sandboxed Python namespace the agent programs directly. Context management becomes a coding task. 86.7% on LOCA_256K, +37.4pp over the prior best long-horizon agent result. Lin, Ang, Zhu, Ding & Zhou, 2026.
The Comparison Nobody Ran Until Now.
When every method including the single-call baseline gets equal optimization effort, multi-call orchestration gains just 4.6 pp over optimized CoT at 2-4x the token cost. No evidence harder tasks justify the overhead. Leins, Pelleriti, Gonnermann-Müller & Pokutta, 2026.
The Missing Baseline That Changes the Math.
Post-training LLMs to use Linear Attention was never compared against SWA with attention sinks, a training-free alternative. When it is, SWA matches on general tasks and wins by 2-10x on long-context reasoning. Jolicoeur-Martineau, Sukthanker, Cameron & Gervais, 2026.
Stop the Run. Learn. Keep Going.
PILOT adds a supervisor that aborts doomed agent runs mid-execution and distills the failure into skills and memory the same run uses immediately. Output tokens drop 43-47%. Successful evals per million tokens rise 110-134%. First place in 5 of 6 benchmark configs. Xiao, Sun, Wu, Hui et al., 2026.
Passing Tests Is Not Migrating.
Agents can copy the original code forward and pass every test without doing the migration. SWE Refactor Bench adds a completeness audit that catches it. Only 5.4% of 520 frontier-model runs survive all three stages. Hong, Chi, Li, Wang, Gao et al., 2026.
Task Success Hides Selection Failures.
Canary tools planted in agent tool sets reveal 36x variance in susceptibility across eight models. Capability tier predicts nothing. Frontier models have one specific blind spot: capability mirages. Anand & Chattaraj, 2026.
The Harness Picks the Winner.
A fragility audit of 12 LLMs under 26 harness configurations finds that config-fragile items carry 95.7% of adjacent-model score gaps, and 4 of 12 models can claim rank one. Benchmark compression makes fragility worse. Parupudi, 2026.
The Environment Is the Bottleneck.
EnvHarness wraps an existing agent training environment in a programmable layer that adapts to the current agent's specific failure modes, without replacing the verifier. Up to 9.0pp held-out gain, 9.8% fewer execution steps. Huang, Wang, Han, Yan, Chen et al., Google, 2026.
Most Agents Never Touch the Algorithm.
AI4AI-Bench asks whether agents can redesign training algorithms, not just tune parameters. Best system: 0.250 (shipping baseline: 0.1). Mechanism-change submissions score nearly 2x parameter-only ones. Reasoning budget moves attempt rate from 8% to 64%. Chi, Li, Hong, Wang et al., 2026.
Run the Frozen Model First.
A frozen untrained model run through a self-improvement pipeline looks like it learned. Seven measurement failures, each inverting a published finding. External distillation has a real effect; self-training does not survive a corrected null test. Xu, Yan, Chen & Kechadi, 2026.
Your Retriever Learned One Dialect.
A tool retriever fine-tuned on one query source style drops from 86.1% to 22.3% coverage when production mixes in other styles. A TF-IDF router with 20 labeled examples per source restores coverage without retraining. Liu, James, Wang, Xiao & Lin, 2026.
Move the Filter, Not the Formula.
The first stage-aware study of context pruning in research agents finds position dominates scoring method. Pre-retrieval filtering cuts tokens up to 73%; lightweight heuristics match trained models; no configuration wins across quality, efficiency, and faithfulness. Kolukuluru, Dernoncourt, Rossi et al., 2026.
Skills Work by Anchoring, Not Teaching.
A study of 8,135 agent trials found skills help by stabilizing action sequences (65.7% of helpful cases), not by supplying missing knowledge (4.5%). The real problem is retrieval: precision collapses from 29.6% at 5 skills to 3.3% at 100. Jiang, Huang, Xing et al., 2026.
The Agent That Knew the Rule.
REDAgentBench checks service receipts instead of transcripts. Trajectory-only grading undercounts violations. In nearly one in five confirmed cases, the agent stated the safety constraint before breaking it. Chen, Liu, Zhu, Dou et al., 2026.
Pass Rate Hides Two Things.
Seven frontier models on 36 long-horizon R&D tasks, graded by what happened inside each run. Agents behave like engineering optimizers: they assemble workable solutions from known techniques, but run-to-run variance is high and experience transfer is near-zero. Final scores hide both. Li, Yang, Tan, Huang et al., 2026.
Your Benchmark May Be Measuring Truncation.
Standard long-context benchmark practice drops content from the middle of prompts. A new paper shows that method removes task-relevant signal alongside filler, making accuracy-vs-context-length degradation curves partly a measurement artifact. Protecting those spans keeps accuracy at or above the full-context baseline. Arjmandi, 2026.
The Decomposer Is the Bottleneck.
Controlled capacity sweeps across a three-role hierarchical search system show where model size actually pays off. Scaling the delegation backbone lifts exact-match accuracy by 11 points. Scaling the execution sub-agents moves it 2.6. A 1.7B distilled executor matches a frontier sub-agent at 37% fewer tokens.
Your Prompt Is the Cost Center.
A preregistered benchmark held task difficulty constant and varied only the words used to ask for help. Open-ended exploration cues inflated deliberation tokens 2.4 to 7.4 times with no correctness gain. A bounded template with scope, acceptance criteria, and a stop condition can cut that spend in half.
Memory Without the Model Call
Researchers asked whether structured agent memory needs LLM generation at all. They organized raw traces as an entity-context graph plus a temporal hierarchy and combined them deterministically per query. Only the final question-answering step calls an LLM. The result: 57.6% lower memory-operation time cost at matched answer quality, with zero LLM tokens on memory operations themselves.
The Retriever Ranks. Something Else Decides.
A retriever tells your agent which tools might be relevant, not how many to actually load. Researchers prove that score-only cutoffs (top-k, threshold) are suboptimal once tool costs are heterogeneous, and show CAM-DF, a training-free stopping rule, cuts tools per query by 37% at matched task success across 1,343 tasks in five tool-use domains.
Interpretability Research Measures the Wrong Thing.
A coalition of NLP researchers proposes a two-question rubric for any interpretability finding: does it name a specific intervention, and was that intervention tested? Most published work fails one or both. The paper identifies five domains where scoring high on both axes produces results behavioral testing cannot.
Not Every Token Needs the Same Compute
Cornell researchers paired a frozen LLM with a tiny policy network that selects attention sparsity, MLP pruning, and quantization bit-width at each decode step. Easy tokens get aggressive reductions; hard ones keep full compute. Up to 7.3-point MMLU gain at matched FLOPs over uniform allocation.
Drowning in Documents
Long-context retrieval collapses not because the window is too small, but because of attention dilution: more documents inflate the softmax denominator and drain probability mass off the gold document. Researchers named the mechanism, built a diagnostic, and showed a 0.6B model can outperform one seven times its size at million-token scale.
Faster. Cheaper. Not Good Enough Yet.
Baidu attached human labor costs to every office benchmark task. Every model they tested was faster and cheaper than hiring a person. None produced deliverable-quality output. Quality-per-dollar is the metric that makes that gap visible.
Fewer Tokens. Larger Bill.
PointFive ran 2,848 metered Claude Code sessions and found that removing 38.4% of tool-output tokens raised billed costs 6.8%. Prompt-cache traffic is ~87% of four-component cost. Compressing outputs addresses the small bucket while the dominant one stays untouched.
Score the Context Before the Agent Runs.
Seven criteria, scored before behavioral tests run, predict which failure modes to expect. In 300 controlled sessions with the model held fixed, moving from poor to structured context cut critical failures 68% (4.11 to 1.33 per session). Context quality is a preflight signal, not a post-mortem metric.
Train the Summary, Not Just the Task.
CompactionRL makes context compaction a trained RL behavior: agents learn what to preserve when the window fills, via cross-trajectory credit assignment that flows across each summary boundary. SWE-bench Verified +7.0pp, Terminal-Bench 2.0 +3.1pp under a fixed context budget.
Let the Models Bid. Route to the Winner.
Agora replaces static agent routing with an incentive-compatible auction: models bid for each reasoning step using calibrated self-confidence, and the mechanism makes honest bidding the dominant strategy. Outperforms single-model, routing, and cascade baselines on five benchmarks.
The Memory Works. The Judgment Doesn't.
MemSyco-Bench isolates a specific failure in memory-equipped agents: deference to stored user statements when those statements should be corrected or ignored. Agents with memory were more sycophantic than no-memory baselines. Most failures were judgment errors, not retrieval errors.
80–90% of Agent Compute Goes to Tasks That Didn't Need It
E3 names the problem (Agent Cognitive Redundancy), measures it (5–10x overspend on simple tasks), and fixes it: classify difficulty first, cap file reads for simple tasks, expand scope only when verification fails. MSE-Bench result: 100% success matched, cost down 85%, tokens down 91%.
Your Workflows Mix Languages. Your Benchmarks Don't.
PolyWorkBench tests agents across 67 workplace workflows where language switches mid-task. A linguistic error at step 2 propagates into every downstream step. The harness accounts for up to 0.21 Pass@1 variance. The LLM judge is essentially uncorrelated with actual correctness at high confidence.
The Memory That Wasn't Yours
FARMA poisons how an agent remembers thinking, not what it remembers. Evasive phrasing bypasses keyword filters. Self-referential amplification bypasses consensus checks. 100% attack success rate against standard defenses. SENTINEL's structural trace analysis closes it to 0% with no false positives.
Don't Let Your Agents Agree Too Soon.
A Shepherd Agent with global context steers isolated Search Agents, each in its own git branch. Context isolation is the diversity mechanism. The swarm outperforms or matches SOTA LLM-guided evolution on 13 of 15 open-ended optimization tasks.
Stop Retrieving. Start Navigating.
NapMem organizes user memory as a four-level pyramid and exposes each level as a callable tool. An RL-trained agent learns to navigate between granularities based on what it has found so far, stopping once it has enough evidence. Better answers, fewer unnecessary tool calls.
The Benchmark That Grades the Journey, Not Just the Destination
UniClawBench runs agents in live Docker containers and scores step-by-step checkpoints across five separated capabilities. The headline finding: framework choice moves scores as much as model choice. Run your base model under two frameworks before blaming the model.
The Loop That Checks Itself
Most self-improving agents have no way to verify they actually improved. SEA adds a frozen base model, a steering adapter, and an anytime-valid statistical gate that controls the false-positive rate at any stopping point. +4 and +5 resolved instances on SWE-bench Verified.
When Agents Reason Well, But Plan Poorly
Step-wise reasoning is a greedy policy. It works on short tasks and collapses on long ones. FLARE adds bounded lookahead and backward value propagation, raising recovery rates from 5.4% to 29.7%. LLaMA-8B with FLARE beats GPT-4o without it.
The Agent That Writes Its Own Playbook
Most LLM agents carry a fixed toolkit forever. This paper builds the system that watches successful runs, notices the recurring multi-step sequences, and promotes them into callable tools. 2.5 to 13.4 percentage point improvement on ACEBench, fewer rounds per task.
When the Agent Says Done, Check the State
78% of a budget agent's failures were tasks it thought it completed. Tools agreed. Database didn't. Four deterministic read-only gates before each tool call recovered most of those failures, +12.4pp on gpt-4o-mini, without touching the model or prompt.
Static Training, Shifting World: Why Tool-Using Agents Break in the Open
An ICML 2026 paper from Nanjing LAMDA formalizes the four ways the open world breaks trained agents: query shift, action shift, observation shift, domain shift. Both SFT and RL training degrade. Perturbation-Augmented Fine-Tuning improves robustness without out-of-distribution data.
Agents Waste 28–64% of Their Budget on Tasks They'll Fail
Frontier LLM agents cannot predict their own failure. BAGEN measures the wastage, finds task skill barely predicts budget-awareness (r = 0.35), and trains early-stop and alert behaviors via SFT+RL. Budget prediction is a metacognitive skill that standard training ignores.
When the AI Asks First, the Answer Is Better
An evidence-first investigator agent that doesn't answer the question you asked -- it first asks whether your premise is right. Scores ambiguity, generates competing hypotheses, asks targeted clarifying questions, and commits only when evidence tips decisively. Beats direct-prompting and reasoning-only baselines.
The Skill Library That Knows When Skills Conflict
Retrieving individually relevant skills isn't enough -- an agent needs a compatible set. Tencent proves bi-encoders can't learn compatibility signals, mines LLM rejection decisions as training data, and adds a reranker that jumps set-level accuracy 13 points.
When Observation Masking Helps Your Search Agent -- and When It Doesn't
A regime map from sweeping 4B-to-284B models across four search benchmarks: masking stale retrieved content peaks at +12.6pp when a strong retriever meets a mid-capacity model, and collapses when the model is already saturated. Don't mask by default; profile your regime first.
The Retriever That Matches by Method, Not Topic
Standard RAG retrieves by semantic similarity. For training reasoning models, that's the wrong signal. RA-RFT trains a retriever to rank by expected reasoning benefit, then fine-tunes via RLVR on analogous demonstrations. Up to +7.1 points on AIME 2025.
The Benchmark Score Is Also a Harness Score
Seven agent benchmarks, one standardized harness, 15 models, 400K rollouts. Scaffold choice and environment volatility move outcomes enough to reorder leaderboards. A unified framework that finally separates model effects from harness effects.
DeployBench: When Agents Say Done and Nothing Runs
Four frontier agents, 51 research artifact deployment tasks, clean environments. Best pass rate: 51%. The dominant failure in 97 of 154 cases: agents that stopped before checking whether anything actually ran.
Evaluate Agents Without Deploying Them
Testing a new agent means deploying it. ADWM breaks that constraint: train a diffusion world model on historical interaction logs, then simulate the environment for any candidate agent before it reaches users. Off-policy evaluation that actually works for LLM agents.
Agent Planning Benchmark: What's Actually Failing in Your Agent?
Most agent evals give you one score and no attribution. APB separates planning failure from execution failure across five settings and 22 domains, then adds a sixth test most teams skip: can the agent recognize when a task is impossible?
ContextPilot: The Prefill You Already Paid For
LLM applications send the same context blocks over and over. RAG pipelines re-attach the same documents. Agents re-inject full histories. ContextPilot intercepts this, reorders shared blocks into cache-reusable prefixes, and deduplicates the rest. Up to 3x faster prefill, 36% fewer tokens, no model changes.
Prompt Injection Has No Complete Defense. Here's Why.
UMass researchers derive an impossibility: any norm that blocks all injections also blocks flows a legitimate task needs. Data-instruction separation is a partial mitigation. Red-team with context-aware attacks, not jailbreak strings.
Beyond Consensus: What Votes Throw Away, Traces Preserve
In mixture-of-agents stacks, majority voting discards the most useful signal: correct intermediate steps buried in minority chains. A trace-reading aggregator recovers correct answers even when every agent voted wrong.
VibeSearchBench: When Queries Get Vague, Agents Fall Apart
A benchmark for how people actually search: vaguely, over multiple turns, revealing intent piece by piece. Best frontier model: 30.3 Triplet F1. Good clarifying questions matter more than retrieval method.
MUSE-Autoskill: The Skill Lifecycle Agents Were Missing
Agent skill libraries don't improve themselves. MUSE-Autoskill builds the five-component lifecycle that lets them: create skills from task successes, gate admission with unit tests, refine from new examples, retire underperformers. Auto-generated skills beat the human-skill ceiling on tasks where generation succeeded.
Is Agent Memory a Database? The Missing Data-Foundations Layer
Vector stores can ingest and retrieve. They cannot revise. Concordia researchers formalize what agent memory actually needs: four operators (ingest, revise, forget, retrieve) replacing CRUD, and argue the missing revise operator is the root cause of most memory failures.
Compute Where it Counts: Per-Token Efficiency
A tiny policy network reads what the model is thinking at each decode step and sets how hard to run that step. 7.3 MMLU points gained over uniform compression at matched compute.
Interpretability Can Be Actionable: A Rubric for the Field
Two criteria for any interpretability finding: does it specify what to change, and has the change been validated? A position paper arguing the field has optimized for the first criterion and underweighted the second.
Remembering More, Risking More: Memory Accumulates Safety Risk
Eight memory architectures, three deployment scenarios: safety violation rates climbed monotonically as memory grew. The same agent with more history produced more violations on identical probe tasks. Risk is detectable before generation.
δ-mem: Adding Persistent Memory to Any Frozen LLM
A compact 8×8 state matrix compresses past context and injects memory corrections into attention at decode time. 1.31× the backbone on MemoryAgentBench. The frozen backbone is never touched.
AgentTrust: A Runtime Safety Layer for Every Tool Call
An 8-component interceptor evaluates every agent tool call before it executes. Deobfuscates shell payloads, tracks multi-step attack chains, proposes safer alternatives via SafeFix. 95.0% verdict accuracy at millisecond latency.
Automated Interpretability: Two Agent Loops, One Autonomous Pipeline
A discovery agent navigates the activation graph to find which features are worth examining. An explanation agent probes and refines hypotheses through contrastive testing. Beats one-shot auto-interp baselines on Gemma-2.
AutoTTS: When LLMs Discover Their Own Reasoning Strategies
A coding agent found a controller that cuts inference tokens by 70% vs running 64 parallel samples. Full discovery cost $39.90 and 160 minutes.
Dual-Dimensional Consistency: Smarter Self-Consistency Sampling
Weight self-consistency votes by reasoning quality, prune weak paths early. Over 10x token reduction at matched accuracy across five benchmarks.
When Weak Models Beat the Frontier: The Boosting Connection
A nano-model committee hit 76.4% on SWE-bench Verified, matching flagship models. The framework explains exactly when and why committees of weak models work: the task needs a local verifier.
Is Grep All You Need? The Harness Matters More Than the Method
Four agent harnesses, same 116 questions, two retrieval methods. The framework around the search tool moved accuracy more than grep versus vector. Claude Code favors grep. Gemini CLI favors vector.
BoundaryRouter: Learning When to Escalate to an Agent
A training-free cold-start router builds experience memory from a seed set. 60.6% inference time reduction versus always-agent, 28.6% accuracy gain over always-LLM.
LaTER: Latent-Phase Reasoning Cuts Tokens 32% Without Losing Accuracy
A training-free two-phase method explores in latent space first, then switches to explicit chain-of-thought only when needed. 32% fewer tokens, better AIME accuracy.
ComplexMCP: Three Failure Modes in Large-Scale Tool Sandboxes
150+ interdependent MCP tools. Three named failure modes. The gap between benchmark performance and production behavior, measured.
STALE: When Agent Memory Becomes a Liability
A 1,200-query benchmark finds the best frontier model scores only 55.2% at detecting stale memories. Implicit conflict is where production agents silently break.
Meta-Harness: The 6x Gap Lives in Your Code, Not Your Model
Varying only the harness around a fixed model creates a 6x performance spread. Stanford and MIT built a system to automatically search for better harness code.
How Coding Agents Actually Perform in the Wild
110,000 open-source PRs, five agents. Code gets merged, but churns faster than human-written code over time.
Conversation Reduces Load. Images Build It.
A 124-person RCT found multimodal conversational AI produces better biology learning outcomes than text-only chat or semantic search.
Image Generation Diversity: When Models Miss the Map
No SOTA image generator covers more than 77% of its training distribution. FID can't detect the gap. IRS measures it. DiADM fixes it.
SLOW: The AI Tutor That Thinks Before It Speaks
A four-stage reasoning workspace that separates cognitive diagnosis from response generation. The tutor thinks before it speaks.
Single-Agent LLMs vs Multi-Agent Systems: Equal-Budget Reasoning
Control for compute and the multi-agent advantage on multi-hop reasoning largely disappears. A synthesis of Tran and Kiela (2026).
LLMs in Games: When Generated Content Runs the Rules
Students embedded LLMs as architectural components in two games. Model errors turned into fairness violations.
Arknights: When the AI Lies, Players Learn
Deliberately unreliable AI guidance produces deeper understanding than transparency could.
The Research Is There. The Understanding Isn't.
24,000 AI papers a month, almost none reaching the people who could use them. Here's why the translation layer matters.
Vibe Coding: Flow, Trust, and Co-Creation
The first qualitative study of vibe coding reveals a paradigm built on flow and calibrated AI trust.
Games That Teach AI Ethics
Two multiplayer games use text-to-image AI to teach teens about bias in generative AI through play.
BAVT: Spend Less, Reason Better
Budget-Aware Value Trees cut AI agent costs by 75% with equal or better accuracy.
What is Vibe Coding?
The emerging practice of building software by describing what you want to an AI.
How to Showcase Your AI Projects
You built something with AI. Nobody can find it. Here's what we learned.
Humans dream it
Agents build it
We ship it together