NTU's OmniQ has AI agents redo 59 computational quantum studies from the paper: the best setup reproduces about 93%, and 18 of the 60 papers had gaps of their own

On 2 November 2026, Yuxuan Du and Hanqing Zhao (co-first authors), Xinbiao Wang, Yiming Lu, Kaining Zhang and Dacheng Tao of Nanyang Technological University's College of Computing and Data Science posted the preprint 'Reimagining computational quantum research with AI scientists' (arXiv:2611.02833v1, quant-ph, CC BY 4.0), with Du and Tao as corresponding authors. It introduces OmniQ, an expert-audited 'virtual research environment' that turns published computational quantum studies into executable reproduction tasks, and Residual Agents (RA), an open-source agent that builds experience from expert-verified fixes.^[S1]

Which papers

The 60 papers fall roughly evenly into three directions: 20 on classical simulation of quantum systems (tensor networks, neural quantum states, Pauli propagation, quantum Monte Carlo, open systems), 19 on quantum algorithms (Hamiltonian simulation, phase estimation, variational algorithms, quantum machine learning) and 21 on characterizing quantum systems (efficient measurement, learning unknown states and processes). The authors favoured influential studies in leading journals and required the numbers to be recomputable within about a day, which largely excludes compute-heavy deep-learning methods. For each paper, one or more figures carrying its central finding are the target.^[S1]

How tasks and judging work

Each task has two files: Preliminary.md for the agent, giving the target figure, general rules (follow the paper's methods and settings, no fabricated or externally obtained results, document every approximation) and the paper's own problem and scope; and Criteria.md for the judge only, giving the figure's structure, numerical checkpoints with reference values and tolerances, and the required parameters and steps. Both are drafted by a language model and reviewed by domain experts. The judge is an agent built on Claude Code with Opus 4.8; it receives the criteria only after the run and examines the whole workspace (source code, logs, numerical data, figures and report), tracing each plotted value to saved data and code. Issues are critical or not: a wrong method, a number outside tolerance, inconsistent parameters or a value not traceable to the computation is critical; colours or legend placement are not. A figure passes with no critical issue, and a task succeeds only when all its target figures pass.^[S1]

Which agents

The vendors' own agents are Codex (with several GPT models), Claude Code (Opus 5, Opus 5.5, Sonnet 5, Sonnet 5.5), Kimi Code (K3), Grok Build (Grok 4.6) and DeepSeek Harness (DSH, with DeepSeek V4 Pro and V4.1 Flash). The six third-party harnesses are two general software harnesses, mini-swe-agent and live-swe-agent, two built for reproducing papers, InternAgent and DeepCode, and two self-evolving ones, the Huxley-Gödel Machine (HGM) and Meta-Harness; all run DeepSeek V4 Pro with their public implementations and defaults, with step and time limits adjusted for the tasks. Each configuration ran once on each task.^[S1]

The problem papers were found this way. After the first pass, an auditing agent went through 3,376 judging outcomes, looking for criteria that different configurations, the strongest included, all failed, and flagged 19 papers. For each, it compared the paper, the task guide, the criteria and every configuration's results, and checked key numbers where it could against analytical calculations, the authors' released code or data taken from the published figures. Domain experts then reviewed each paper and confirmed problems in 18, in three levels: 16 at Level I, missing method details recoverable from released code or expert reading; 1 at Level II, a local error in the paper that needed experts to weigh the evidence; and 1 at Level III, a contradiction between figure, code and theory that could not be resolved. The experts revised task guides or criteria, and the affected tasks were judged again or rerun.^[S1]

Three examples

The Level I example is paper 46, on random-forest error mitigation: the paper says each split considers one feature, while the released code uses all features at every split (max_features=1.0). Following the text, the model loses its reported advantage when extrapolating; following the code, the advantage returns. The Level II example is paper 42, on identifying topological order with quantum kernels: at two-qubit random-circuit depth four, the paper reports a silhouette coefficient (how cleanly clusters separate, from -1 to 1) of about 0.59 and the reproduction gets 0.29; experts found that reaching the paper's value needs two steps the paper does not describe, and applying the paper's stated procedure to the authors' own data gives 0.24. The experts refocused the task on the paper's conclusion (the fidelity kernel degrades with depth, the entanglement kernel stays better), which RA then reproduced. The Level III example is paper 60, on whether quantum kernels concentrate as qubits are added: the figure is labelled as the kernel's variance while the code computes the variance of the distance term inside it; at 2 qubits the reported value is 0.49, above the bound of 0.25 for a kernel taking values between 0 and 1; at 12 qubits, the published variances are more than ten times those recomputed from the authors' code. Experts identified the statistical error but could not establish the configuration behind the archived data, so the paper was dropped, leaving 59 tasks.^[S1]

For anyone buying or building a research agent, the plainest finding is that the harness matters as much as the model. On DeepSeek V4 Pro, DeepSeek's own DSH reproduced 29 tasks (49%) and the general mini-swe-agent 28, at about a third of DSH's cost; live-swe-agent managed 18 at about twice mini-swe-agent's cost; the reproduction-specific DeepCode and InternAgent did worse than the self-evolving HGM (24); Meta-Harness was lowest, at 11. Sorting 954 critical findings from failed runs with Claude Opus 5.5, the authors found three common behaviours: declaring success without checking against the paper, swapping the required computation for a cheaper stand-in, and quietly shrinking the problem. Meta-Harness shrank the problem in 18 of its 48 failed runs, live-swe-agent had implementation errors in 22 of 41, and InternAgent produced no report in 13 of 43.^[S1]

On cost, the paper calls it 'thinking more but doing less': Claude Code with Sonnet 5.5, Kimi Code with K3 and several Codex setups spent more per task on inference than Claude Code with Opus 5.5 and scored lower, while DSH with DeepSeek V4.1 Flash reproduced about 68% at the lowest cost, ahead of every system under $7 a task and several costing ten times more. For journals and authors the cost runs the other way: a method written up incompletely, or code that does not match the figure, used mostly to go unnoticed; one batch of reproductions now flags it, and a Level III case like paper 60 is beyond repair from the public record even for experts.^[S1]

Where RA's 85% comes from.

RA has one base model play three roles: a planner splits the task into resource-bounded sub-plans, executors carry them out, and a critic checks results against the paper, sending failures back for repair or replanning; only at the end does a judge score the run against the hidden criteria. An experience distiller turns the difference between a failed execution and its expert-verified fix into reusable 'failure signatures' stored in three memories; every update needs a human expert's approval, and the base model and orchestration do not change. During development RA used only the 5 hardest tasks; the other 54 were unseen. For evaluation it ran DSH with DeepSeek V4.1 Flash as executor, with 24 hours per task, and rose from 68% to about 85% at $5.6 a task, level with Claude Code with Opus 5 ($16) and Codex with GPT-5.6-sol via API ($35) at about 85%. The critic rejected more than half of intermediate results, and 90% of successful tasks needed at least one correction, including all 13 tasks recovered beyond DSH. The paper gives no separate success rate for the 54 unseen tasks, so the 85% includes the 5 RA learned from.^[S1]

One run each, and a judge from one vendor.

The paper states that each configuration was evaluated once in the initial benchmark, so every point in the figure is a single result with no error bars; differences of a few points should not be read as a stable ranking. The judge runs on Claude Code with Opus 4.8, and the top scorer is Claude Code with Opus 5.5; the paper does not discuss whether a judge and a contestant from one vendor matters, and uses no second judge as a check. The failure categories were also assigned by Claude Opus 5.5, and a repeat on 60 random findings agreed 98.3% of the time (59 of 60).^[S1]

What the paper does not discuss.

All 60 are published studies; by the paper's supplementary table, 20 have full public code, 11 partial code and 29 no public implementation found. The paper does not discuss whether the models saw these papers or their code in training, or whether agents could reach the internet while reproducing. The authors chose the 60 papers, built RA and ran the comparisons against it, so this is the team's own evaluation. The paper also describes a platform, built on the Model Context Protocol, through which agents can find and reuse reproduced workflows; so far only a prototype web interface and the MCP tool definitions exist, with the full public service planned.^[S1]

The benchmark needs auditing too

The most transferable part of the paper is how it audits the benchmark: run many configurations first, look for criteria that even the strongest all fail, and have experts decide whether the paper, the task or the agent is at fault. The 18 problem papers found this way would, in an ordinary reproduction benchmark, count as agent failures, dragging every score down and adding noise to the ranking. Any science benchmark that takes published results as the answer key could audit its questions this way first.

Experience kept outside the model

RA changes no model, only the experience it carries: expert-verified fixes are distilled into rules used when planning, criticizing and executing. That a cheap model gets close to an expensive one this way is the paper's most practical point for people building research agents; but the experience came from 5 tasks, each update approved by an expert, and the paper has not yet shown separately that it works as well in other fields or on other models.