An alphaXiv-led study runs 936 AI agents from 35 labs as one research community for twelve weeks, and they publish 936 computational studies with code and data
Rehaan Ahmad (alphaXiv), Samuel Schmidgall (UCSF) and colleagues report that 35 human-run labs deployed 936 autonomous agents on a shared preprint server from August to November 2026. The agents published 936 studies with code and data, sent 9,500 messages and started their own peer-review journal. An agent audit traced 23,811 reported numbers to output files: 93.8% matched exactly and the rest differed only in rounding or format. Two LLM judges rated agent papers above matched human arXiv preprints on rigor and below them on novelty.
What happened: Rehaan Ahmad (alphaXiv), Samuel Schmidgall (UCSF) and colleagues report that 35 human-run labs deployed 936 autonomous agents on a shared preprint server from August to November 2026. The agents published 936 studies with code and data, sent 9,500 messages and started their own peer-review journal. An agent audit traced 23,811 reported numbers to output files: 93.8% matched exactly and the rest differed only in rounding or format. Two LLM judges rated agent papers above matched human arXiv preprints on rigor and below them on novelty.
Why it matters: It records hundreds of AI agents doing research as a community over three months, not one agent on one problem, and it separates where their output can be trusted from where it cannot. Numbers checked against code and output files matched; prose did not: 68% of 552 null-result abstracts called a non-significant result proof of no effect. Text-only AI reviewers missed flaws, while turning 156 major objections into scripts run on the data overturned the headline in 77. The authors' lesson for anyone using agents in research: verify against executable artifacts, not text.
Evidence and limits: All studies were computational, with no wet lab; local model experiments mostly used open models of at most 14 billion parameters. Cohort agents ran the audits, which check that numbers match saved outputs, not that experiments were valid; the clinical and solar audits are not yet expert-reviewed. The human comparison used two Gemini models as judges, and the agents' overall edge came entirely from Lab 1. The ban on human coding relied on participants' own disclosure. A preprint.
































Commentary
1,900 AI-for-science papers in one week, five questions a test errorcannot answer
Jorge Bravo-Abad argues that a small error on held-out data does not show a model is right about what it was built to predict. Changing only the labels' dispersion correction put a perovskite's last phase transition more than 150 K below experiment; one force field's error more than doubled on molecules it had not seen.Read more
Argument: Jorge Bravo-Abad argues that a small error on held-out data does not show a model is right about what it was built to predict. Changing only the labels' dispersion correction put a perovskite's last phase transition more than 150 K below experiment; one force field's error more than doubled on molecules it had not seen.
Why read: Five controls anyone can run on their own model: vary the label source, compare the error with the energy scale of the question, report error by distance from the training data, test against a sham or a null case, and compare a generated ensemble with the runs it was trained on.