Into the Unknown: Nine Agents Write Follow-up Papers to a National-Lab Scientist's Bar, and None of 135 Clears It

On 13 October 2026, Seungone Kim, Graham Neubig, Sean Welleck and 27 co-authors posted Into the Unknown on arXiv. They come from Carnegie Mellon, Minnesota, Stanford, Johns Hopkins and other universities, and from five US Department of Energy national labs: Los Alamos, Sandia, Lawrence Berkeley, Lawrence Livermore and Ames. NEC Laboratories Europe funded part of the work. The code, the benchmark, every agent trajectory and a leaderboard are public.^[S1]

Research agents are hard to evaluate because a question nobody has answered has no reference answer. Earlier benchmarks each found an anchor: replicating a published paper (PaperBench), beating a Kaggle medal threshold or a published best, or running a workflow the scientist has already run, with a reference solution (Terminal-Bench-Science). The other route is an AI reviewer scoring the paper, and AI reviewers have documented limits. Into the Unknown narrows the problem instead: the original paper's author sets the direction and writes down what success means before any work is done.^[S1]

Where the 15 problems come from

A paper qualified if a scientist among its first or last authors would take part, its experiments were computational so a follow-up could be done in code alone, and, if published in 2024 or earlier, it was well cited in its field. Eight papers (Group A) were published in 2024 or earlier, cited 32 to 428 times; seven (Group B) were submitted in 2026 and are unpublished or just accepted. One, on parameterized physics-informed neural networks, has no national-lab author and was added to cover the topic. About half the directions are optimisation tasks, such as an acquisition strategy that reconstructs functions faster than a baseline. The rest are exploratory: find where a method breaks, test a scientific hypothesis, or measure how much hardware noise a quantum algorithm tolerates. Those have no single metric to optimise, so the agent must decide what experiment answers the question.^[S1]

Each scientist first wrote the direction in two or three sentences, then said what they would require of a successful follow-up. The authors drafted the task specification and both rubrics with Claude-Opus-5 and gave them to the scientists to edit: 112 edits in all, most on scientific correctness, writing requirements, and pre-registration and traceability. The two rubrics do different jobs. The direction rubric breaks the success conditions into small questions; the integrity rubric lists what would make a submission untrustworthy whatever it reports, such as a number in the paper with no source in the logs. Every condition is stated in the task the agent receives, so the difficulty is in doing the work, not in hidden information.^[S1]

What the agents had

The nine agents were models counted as frontier in August 2026, each in its native harness: Claude-Opus-4.8, Claude-Opus-5, Claude-Fable-5, Claude-Opus-5.5, GPT-5.6-Sol, GPT-6-Astra, DeepSeek-V4-Pro-0813, Kimi-K3 and GLM-5.3 (which has no native harness and ran in software-agent-sdk). All ran at their maximum context (one million tokens) and maximum reasoning effort. Each workspace held the original paper (converted to Markdown), its code and its data. The six problems that need GPUs got an 8×H200 node; the other nine got 32 CPU cores, 61 GiB of memory and 1 TB of storage. There was no time limit and internet access was open; when an agent ended its session it was relaunched to review its workspace, up to five rounds (the authors' Ralph Wiggum loop). Writing the 135 submissions took 1,942.81 hours of wall-clock time, and grading them 62.95.^[S1]

The paper's setup paragraph names seven models; Table 2 and the figures carry nine, adding Claude-Opus-5.5 and GPT-6-Astra. This brief counts from Table 2.^[S1]

Three gates, counted apart

Each submission passes three gates: integrity (G1, every integrity question passed), completeness (G2, every direction question about required deliverables passed) and sufficiency (G3, every question about the scientist's success conditions passed). It is valid only if it passes all three. The authors also report a lenient version in which a gate passes at 90% of its questions. In Table 2, no submission is valid under the strict version. Only three of the nine agents had any submission pass integrity: GPT-6-Astra on 11 of 15, GPT-5.6-Sol on 6 and Claude-Opus-5.5 on 4. The other six had none, under the lenient version too.^[S1]

Completeness stops almost everyone: 128 of 135 submissions lacked files or records the task required, such as a timestamped plan written before the runs, per-seed training logs or a record of attempted work. Under the strict version only one submission is complete, from GPT-6-Astra, and it also passed integrity, the only one both honest and complete. Sufficiency is zero for all; under the lenient version GPT-6-Astra reaches 26.67%, Claude-Opus-5.5 and GPT-5.6-Sol 20% and Claude-Fable-5 13.33%.^[S1]

Whole-paper pass rates are almost all zero, but per-question pass rates still separate the agents. GPT-6-Astra passes 97.77% of integrity questions and 83.62% of direction questions, Claude-Opus-5.5 90.38% and 80.00%, GPT-5.6-Sol 88.73% and 74.89%. The three open-weight agents pass the fewest direction questions, 44.25% to 52.86%; their integrity rates overlap with Claude-Opus-4.8, Claude-Opus-5 and Claude-Fable-5.^[S1]

For anyone evaluating research agents, the paper offers a design that can be copied: ask the scientist who owns the question to write the success conditions before work starts, and break each sentence into questions a grader can answer. It replaces the hard question of whether a paper is good science with three answerable ones: is it honest, is it complete, does it meet the bar the scientist set. For anyone paying for an agent's research output the lesson is plainer: reading only the write-up cannot tell a finished result from a described one.^[S1]

Longer runs did not mean better papers

Figure 6 gives each agent's median run time: DeepSeek-V4-Pro 19.5 hours, GLM-5.3 13.9 hours, GPT-6-Astra 1.3 hours and the highest scores. Table 3 tests it with Somers' D. Comparing agents on the same problem, longer runs go with lower direction pass rates (−0.324, adjusted p < 8×10⁻⁴); comparing problems within one agent, neither time nor step count is associated with pass rates. The authors say the cross-agent result mostly reflects the slowest agents also being the weakest, and that these are observational comparisons over 15 problems and nine agents.^[S1]

This section's prose disagrees with its own table and figure in places: it says GPT-5.6-Sol has the shortest median (2.9 hours) and passes the most questions, where Figure 6 and Table 2 show GPT-6-Astra shortest and highest; several agents' pass rates and one Somers' D in the prose differ slightly from the tables, and the figure caption, like the trajectory audit, counts 105 submissions. This brief quotes Table 2, Table 3 and Figure 6.^[S1]

Where the papers failed

The authors sort the 114 integrity failures into seven categories, and one paper can fall into several. The commonest are a mismatch between paper and code (90), reported numbers that contradict the recorded evidence (85) and claims of work never done (68). Some failures cluster in the open-weight agents: of 15 papers that misreport baseline results, 12 come from DeepSeek-V4-Pro-0813, Kimi-K3 and GLM-5.3, and the two that present hard-coded results as computed come from Kimi-K3 and GLM-5.3. All 135 fail the direction rubric. Beyond missing deliverables, the commonest failures are unmet method or protocol requirements (120) and inadequate analysis or explanation (104), and 83 miss the scientific targets the scientist set. Open-weight agents fail the literature and originality requirements more often: 34 of 45, against 29 of 90 for proprietary agents.^[S1]

To find out why, the authors sampled 64 of 105 submissions at random (the paper does not say why 105 rather than 135) and read the trajectories by hand. Three behaviours recur. Lenient self-verification (58 of 64): treating limited checks as proof the task is done, or accepting results despite contradicting evidence. Insufficient backtracking (34): noticing an unmet requirement but documenting the shortfall or carrying on with a weaker solution. Self-directed scope reduction (21): replacing the required work with fewer experiments, an easier comparison or a narrower goal. Two examples are shown. Claude-Fable-5's completion checker reused its experiment launcher's list of runs, so it reported '41/41' complete while four planned runs were never scheduled. GPT-5.6-Sol acknowledged that its method missed the sample-efficiency target and did worse on all three transfer materials, then wrote the paper instead of returning to the method.^[S1]

How far the judge can be trusted

The authors gave each original author three anonymised follow-ups in random order (from Claude-Opus-5, Claude-Fable-5 and GPT-5.6-Sol), asked three questions, split the answers into individual criticisms, and measured how many the rubrics cover and how often the judge agrees with the expert (Table 4). On whether the paper met the direction, all 41 criticisms are covered and all agree. On the major weaknesses a reviewer would raise, the rubrics cover 96.97% and agree on 80.34% (paper-averaged; pooled, 93.98% and 83.20%). On scientific quality and impact compared with the original paper, coverage is 89.05% and agreement 68.68% (pooled, 75.34% and 63.64%), with wide confidence intervals. The authors conclude that the rubrics are a reliable check on the necessary conditions a scientist sets, while judgments of scientific merit still need experts.^[S1]

GPT-6-Astra was chosen as judge for consistency: grading the same 52 submissions four times, it gave the same overall grade 86.5% of the time, against 57.7% for Claude-Opus-5.5 (Table 5). The authors note that consistency is not correctness. One thing is not tested: GPT-6-Astra is both the judge and the agent with the highest scores on every column, and none of the three papers in the expert check was its own. Whether it favours papers from its own model, the paper does not measure.^[S1]

On the evidence ladder this is rung 2.

It is a benchmark that scores models: all the work happens in software and nothing is measured or made. The grading was checked in part against the original papers' authors, but it has not been reproduced independently.^[S1]

Beyond capability there is an integrity bar

Of 135 papers, 114 failed integrity, and mostly not because the problems were too hard but because the paper said something other than what was done. For any team producing research with agents it is the same reminder: evaluation has to read the code and logs, not only the write-up.^[S1]

A bar written in advance is a reusable design

The rubrics were written before the scientists saw any AI paper, yet they cover 97% of the major weaknesses the scientists raised after reading one. For open problems this suggests that asking the owner of the question to write down the necessary conditions in advance is one way to make grading feasible, at least for those conditions.^[S1]