Same Model, Different Harness: What Three Studies Show About the System Around a Model

This brief reads three studies together: VISTA from Kaiming He's group at MIT (a tech report of 1 October 2026, in game environments), BiomniBench from Phylo, Stanford and others (a bioRxiv preprint of 12 May 2026, v2 posted 18 May, on biomedical data analysis), and Periodic Labs' research post of 15 September 2026 (XRD phase analysis). Each runs one model in different surrounding systems on the same tasks, and each reports a large gap. This site has a brief on VISTA and one on Periodic; this one asks only what the three show together, and what they do not.^[S2][S1][S3]

What 'harness' means here

All three say harness, meaning the layer outside the model: what tools it gets, how context is managed, where it executes, what it remembers, what data it can reach. But each changes a different part. VISTA changes how the model sees and remembers: image input, every frame kept as it was, zoom and replay on demand, pixel reads, plus two notes. BiomniBench compares four off-the-shelf general coding agents (Terminus-2, Claude Code, Codex CLI, Gemini CLI) on the same tasks, data and container. Periodic's own harness adds the company's internal structure and simulation databases and lab context on top of tools, while the comparison gets only open databases and software.^[S2][S1][S3]

In BiomniBench, every model does better in its own harness

Table 1 puts nine models in the same Terminus-2 harness: the top four, Opus 4.7 (63.94), GPT-5.5 (63.57), Opus 4.6 (63.16) and Sonnet 4.6 (62.35), sit within 1.6 points, and open-weight GLM-5.1, Qwen 3.6 and Kimi K2.6 trail by 3 to 5. Table 2 puts six models in their own vendor's harness: Opus 4.7 scores 73.34 in Claude Code (+9.4), Opus 4.6 69.51 (+6.4), GPT-5.4 68.69 in Codex CLI (+13.5), GPT-5.5 67.59 (+4.0), Sonnet 4.6 66.39 (+4.0) and Gemini 3.1 Pro 49.83 in Gemini CLI (+5.6). Each setup ran three times, and the standard deviation across the three means, 0.3 to 1.8 points, is smaller than these gaps.^[S1]

Reasoning effort was not matched everywhere

BiomniBench's Appendix A.1 states that Anthropic's models ran in Terminus-2's adaptive variant at effort=max, and in Claude Code also at effort=max, so the Opus and Sonnet gaps are at equal effort. The other models ran in Terminus-2 at reasoning_effort=high (the most that harness supports) and in Codex CLI at xhigh, with Gemini CLI on a dynamic thinking budget (thinkingBudget=-1). The paper calls GPT-5.4's 13.5 points 'attributable entirely to agent architecture', but the effort differs too. The harness also changes how the run goes: GPT-5.4's median turns go from 13 to 82, GPT-5.5's cost per task from $1.98 to $4.58 and Opus 4.7's from $0.87 to $2.52; the highest-scoring setups are also the most expensive.^[S1]

Who scores

VISTA uses ARC-AGI-3's official RHAE formula. BiomniBench has an LLM judge score each task against its expert rubric, criterion by criterion; the authors compared five candidate judges on 597 criteria from 35 tasks and chose Gemini 3.1 Pro, which agreed best with experts (82% per-criterion accuracy, linear-weighted κ 0.70), and recommend human review for high-stakes use. Periodic uses an LLM-judge ensemble of Opus 5 and GPT-5.6 Sol calibrated against expert ratings, on FrontierXRD, 134 unpublished samples from its own labs. The three scores use different units, so this brief compares gaps within each study and never numbers across studies.^[S2][S1][S3]

What the three support together

The firmest conclusion is the direction: across games, biomedical data analysis and XRD, the same model moved a long way when its surrounding system changed, and in BiomniBench, the only study with a generation comparison, the harness gap at equal effort (Opus 4.7's 9.4 points) exceeds the gap between adjacent model generations (3.8). So 'model X scores Z on task Y' is unclear without the harness, the effort and the budget. That stands firmer than any one study, because the three differ in how they compare and what tasks they use, yet point the same way.^[S2][S1][S3]

What they do not yet support

They cannot yet say how much the harness design itself contributes. VISTA's comparison with the official baseline mixes in access route and limits; BiomniBench's largest gap mixes in reasoning effort; Periodic's mixes in private databases and is company-run. None has been reproduced by an independent group; VISTA tested only public games, BiomniBench's tasks are adapted from published papers (agents were barred from reading them, and spot checks found no shortcuts, but memory cannot be ruled out), and only Periodic used samples never published. Nor is there one shape of a good harness: VISTA's gain comes from keeping raw observations, Periodic's from data access and domain software, and BiomniBench's authors argue that no off-the-shelf harness was built for biomedicine, leaving room for one that is.^[S2][S1][S3]

What it means for AI for Science

For anyone evaluating science agents, it means reporting the harness as part of what is tested: model, harness, effort, budget, data access, and without any one of them the number cannot be compared with anyone else's. For teams building science agents, it means investing in the harness may pay off sooner than waiting for the next model, but at a cost: BiomniBench's top setups cost the most per task, and VISTA uses 30.7 million tokens a game. Periodic builds its harness as a core system and runs all its model comparisons on it, which shows at least one frontier AI4S company already works this way; but its 3.8x is the least controlled of the three.^[S1][S2][S3]

Three kinds of evidence would turn the direction into a conclusion. First, controlled substitution: an outside group fixing model, effort and budget and swapping one harness component at a time (memory, tools, context management, data access) on scientific tasks, as VISTA did on games. Second, unseen tasks: ARC-AGI-3's private games, unpublished experimental data, or analyses that appear after a model's release. Third, independent reproduction: BiomniBench's dataset is public on Hugging Face, so other groups can retest with their own harnesses; Periodic's result waits on FrontierXRD going public or an outside retest.^[S1][S2][S3]