Fifteen of Sixteen Agents, One Shared Error: Why Agreement Between Replicas Is Not Independent Confirmation

The preprint is by Jihan Kim of KAIST's Department of Chemical and Biomolecular Engineering, sole author, submitted to arXiv on 21 September 2026. The charter asked for the best validated material and a defended judgment about the database ceiling. The study asks not whether an agent completes a task but how routes, conclusions and failure modes are distributed when the same open-ended research mandate is handed to several agents.^[S1]

The task is methane storage in porous materials. Deliverable capacity (absolute loading at 65 bar minus that at 5.8 bar, at 298 K, converted to cm³ of methane at STP per cm³ of framework) is a clear practical figure of merit, standard GCMC is cheap enough for week-long campaigns, and the literature maps the landscape well: credible leading structures fall near 190-200 cm³/cm³ under standard conditions, so a value well above 200 deserves scrutiny.^[S1]

The database is the computation-ready CoRE MOF 2024 set, frozen at 12,499 entries and pinned by a SHA-256 manifest; after merging coordinate-identical entries, 9,167 distinct structures remain. Every agent used the same RASPA build, TraPPE methane model and UFF force field, and could not alter the protocol behind a final claim. Each received 1,610 CPU-hours (against an estimated 22,873 for naive exhaustive screening), 32 million tokens and a US$280 spending limit, which was the binding constraint for twelve of sixteen.^[S1]

All sixteen agents were separate sessions of one frontier model, recorded only by the alias claude-opus-5 with no finer snapshot identifier, run through Claude Code 2.1.233 between 29 August and 1 September 2026. Eight were randomly assigned an appendix of seven enforceable checks: two capacity thresholds, structural sanity, protocol validity, modification hygiene, finalist reproduction and random auditing; the other eight got the same scientific norms without enforcement. Both groups were told to investigate suspicious results and prefer a correct negative conclusion over an inflated record. No agent was told other agents or groups existed.^[S1]

Routes separated right after the first step

All sixteen began by submitting a simulation job. After that, every agent's first five logged actions formed a unique sequence, and the campaigns resolved into four approaches: descriptors plus surrogate models, fast low-accuracy coverage, serial progress on one or two cores, and treating the database as a starting point for design. Reported screening ranged from 106 to 5,006 structures, a nearly fiftyfold spread.^[S1]

Even so, conclusions converged. Setting aside the audit-excluded entry, twelve agents named at least one audit-retained material above 180; eleven reported the same copper-based pts framework, ten placed it first among retained materials, and they supplied ten separately generated high-accuracy values between 198.9 and 200.1. Vanadium srs, ytterbium nia and indium nuc frameworks followed.^[S1]

The author frames the problem at the outset: replicas of one model may follow different routes while sharing the same database, software and scientific assumptions, so their convergence may indicate a robust result, a common scientific prior or a common-mode error.^[S1]

What the shared error was

Fifteen of sixteen agents named the same copper sql framework, 2021[Cu][sql]2[ASR]6 (its [FSR]6 file is coordinate-identical), as the best material in the database, reporting 206.7-207.2 cm³/cm³ with a standard deviation of only 0.12, below the reported simulation uncertainties. Seven checked and all eight unchecked agents reached it; the remaining agent did not spot a problem: it measured a different deposition sharing the same identifier stem as a cost benchmark and never ran the excluded file at any fidelity.^[S1]

The value was reproducible; the object was wrong. The entry derives from BSF-71, a copper-dipyridyl framework whose square-grid sheets are charge-balanced by non-coordinating [B₁₂Cl₁₂]²⁻ anions held between the layers, with acetone on the copper axial sites. The published crystal structure contains the anions and solvent; the database entry retains only Cu(dpb)₂, the anions having been removed together with the solvent during curation. The vacated interlayer space appears as pore volume, and the entry's simulated capacity exceeds the leading retained entry by about 7 cm³/cm³.^[S1][S4]

The author notes that missing counter-ions and charge imbalance are documented curation problems in computation-ready MOF databases, which sharpens rather than softens the observation: sixteen agents with that literature available treated a known class of artefact as a discovery. Neither the model nor the simulator invented or miscalculated the capacity; every agent inherited the same incomplete database object.^[S1]

How it was caught: not by the agents

The error was caught before launch, by the investigator. A three-pass charge-accounting procedure, first validated on 70 ZIF-like structures that returned net charge exactly zero, flagged 406 compositionally charge-inconsistent entries as a lower bound. Most were too dense to affect a methane screen; six combined the imbalance with anomalously large void space and were excluded prospectively as capacity artefacts under mechanical rules recorded in the answer key. The agents received neither the flagged list nor the audit tool.^[S1]

The agents had the clues. At least ten recorded signs that the champion was unusual (bare copper centers, low density, very high void fraction, disconnected two-dimensional sheets, extreme surrogate-model residuals), and three checked agents logged the bare copper centers under a caveat their appendix required. Each sign was read as an explanation for exceptional performance rather than a reason to question the structure. One checked agent built a chemistry audit and correctly calculated a +8 charge across the four copper centers, then balanced it with negative groups absent from the file, importing expected chemistry into the deposited composition. No agent validated a new chemistry audit against independent known cases.^[S1]

The enforced checks did change behavior: all eight checked agents re-derived their headline numbers from saved inputs against one of eight unchecked (Fisher exact test, p = 0.001), and five verification measures totalled nineteen acts against nine (Mann-Whitney, p = 0.012). No improvement in conclusion validity was detectable: one checked agent named an audit-retained champion, and only because it never examined the excluded entry, not because it recognized the defect (p = 1.0). The author adds that with fifteen of sixteen naming the same entry the design had essentially no power to detect a difference, so not detecting one does not show the checks had no effect. In the author's words, the checks did not target the layer at which the decisive error arose. One checked agent reproduced the excluded entry's value with a fresh random seed to within 0.05 of its reported uncertainty and passed every enforced audit.^[S1]

The author makes the distinction concrete: for a random sample of three agents, there was a 98% chance that at least one would report the leading retained material, and certainty that a majority would name the excluded entry. Replication rapidly increased confidence in both the frontier and the shared error. Agreement across repeated agents establishes stability to resampling of one model and workflow, not independent confirmation by different data, methods or scientific perspectives.^[S1]

The error also propagates. Three modification programs adopted the apparent champion as a parent or benchmark, letting a screening result become an organizing assumption for downstream work within the same week; one agent compressed its lattice by 4% and raised the value to 214, though it described the products as evidence about the protocol's ceiling rather than as materials.^[S1]

The evidence stops at computation, but the check was fixed in advance.

The agents' claims were scored against a hidden answer key and scoring rubric, sealed with the charters, group assignment and analysis plan by a public-hash commit before launch: the paper's pre-registration. The key combined the pre-launch charge audit, a literature-calibrated ceiling and an independent reference calculation. The full-database reference was specified before launch and is still running; the interim analysis behind Fig. 4 was defined in an amendment filed on 3 September, before any report was read. It measured 2,344 audit-retained structures under the pinned protocol: 1,497 drawn uniformly at random from the 9,167 distinct structures, plus 847 from a high-porosity tail selected by a rule fixed before the calculation. The nine highest point estimates are all materials at least one agent reported, from Cu pts at 199.4 ± 1.3 down to 190.3; the first structure no agent named is tenth, at 190.2. The author notes the uncertainties overlap within this group, so the ranking is by point estimate, not a resolved ordering.^[S1]

Why rung 3.

The result was produced in software and checked against a reference and a database audit fixed in advance, the audit resting on the published crystal structure; nothing was synthesized or measured, and no modified structure was relaxed or checked for stability. The reference and the audit are the author's own, so the check is internal.^[S1]

Limits the author states.

The results characterize one model-harness, database and task, not a universal agent failure rate. The model appears only under the alias claude-opus-5, so identical weights across the campaign cannot be guaranteed and the study cannot be rerun on a pinned snapshot. The full-database reference census continues: the random sample covers about 16% of structures outside the tail, so frontier members invisible to the descriptor rule may exist unmeasured. Workspace isolation was a rule audited through records, not an operating-system boundary, and one shared-directory failure delivered foreign files to two workspaces; both detected and quarantined them, no reported value was affected, and excluding both leaves the main conclusions unchanged. Four exploratory behaviors totalled ten in the checked group against seventeen unchecked, not significant (p = 0.14): an observed trend toward less exploration, which the author says suggests but does not establish that the checks displaced exploration under a fixed budget.^[S1]

It can be checked.

Code, the answer key, the pre-registration, the analysis scripts and all sixteen agent repositories with complete git histories and verbatim transcripts are in a public repository, with the submitted version archived on Zenodo. Structure files are distributed as a manifest of identifiers and hashes rather than coordinates, because part of the set derives from the Cambridge Structural Database.^[S2][S3]

Anomalies were read as performance

The most useful detail here is not 15 of 16 but the ten agents that recorded anomalies and did not question the structure. Anomalies prompted several agents to question and rerun the calculation, yet none led them to question the deposited structure itself. The author's conclusion: effective checks must test whether the object under study is valid, not merely whether the calculation is reproducible.^[S1]

Validate the audit tool itself

The author's charge audit was validated on 70 ZIF-like structures with a known net charge of zero before it was applied to the database; none of the agents' home-built chemistry audits was validated against known cases, and the one that correctly found a +8 charge balanced it away with groups absent from the file. Audit the database before the agents run, and run the audit tool on known answers first.^[S1]

Three quantities to keep apart

The author argues that task-success evaluations can conflate three quantities: diversity of research trajectories, stability of conclusions and independence of the scientific substrate. Here the first two were high and the third close to nil. Diverse routes are not independent evidence.^[S1]

What it means for reading agent claims

A growing share of AI-scientist work backs its conclusions with agreement across runs or agents reviewing each other. This paper supplies a counterexample to check against: when every replica reads the same data, agreement shows the result is stable under the same setup; it cannot rule out an error they all share, such as one in the data. The question to ask of such a claim is where the confirmation comes from: other data, another method, another model, or the same model run again.