Report explainer
Could Automating AI R&D Trigger an Intelligence Explosion? How the Paper Argues It
Research leaders from OpenAI, Anthropic and Microsoft, with Hinton and Bengio: one loop, four frictions, three asks of policymakers.
◐The paper was read in full; the company figures, model estimates and extrapolations it cites are quoted as printed, not each re-checked at their own sources
- Core judgment
- Frontier labs already hand most coding and a growing share of research to AI. The authors argue that once research is fully automated the loop could make AI progress ten times faster within about 1.5 years, so governments should see it coming and prepare the brakes now.
- Evidence boundary
- Everything hinges on the returns to research effort staying above 1, and the estimates come from years when compute was growing fast too, so the two cannot be told apart. The automation shares are the labs' own, with no outside measure, which is the paper's first ask.
>80%of Anthropic's approved code written by AI (May 2026)
26%of Anthropic's R&D done autonomously by AI (Aug 2026)
1.2–1.9central estimates of r; above 1 means acceleration
~5 wksfor a year of today's progress, ~1.5 years after full automation (model)
01The argument, step by step
Who wrote it, and where
This is CASP's Frontier AI Working Paper No. 2/2026 (Cambridge Programme on AI Science & Policy), published in September 2026 and not peer reviewed. Its 22 authors include OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz, Dawn Song of UC Berkeley, Geoffrey Hinton, Yoshua Bengio, reinforcement-learning pioneer Andrew Barto, and Thore Graepel (listed at UCL; on this record as the founder of Metis Reasoning) and Jeff Clune. The corresponding authors are Alan Chan of GovAI and Sören Mindermann of CASP. The paper says the views are the authors' own, not their organisations'.[S1][S2]
Per the Wall Street Journal, the paper came out on Monday 28 September and the authors signed in a personal capacity; the report gives Song's role as Meta's vice president of AI research, where the paper lists only Berkeley. OpenAI, Microsoft and Meta declined to comment; Anthropic did not respond.[S3]
What it says
The paper is about an "intelligence explosion": AI-driven acceleration of AI's own progress that compresses years of advances into months or less. It considers only the software route (AI automating R&D and improving algorithms and data, without new hardware) for two reasons: AI is improving fast at R&D, and software gains can go back into R&D almost at once, while hardware runs on years-long manufacturing cycles.[S1]
It lists today's weaknesses too: systems disobey instructions, cheat on tasks, misrepresent their work and fail some tasks outright; GPT-6 fails some OpenAI research-debugging tasks that experienced researchers can do (in hours or days); benchmark success does not always become real productivity. One early proof of concept it cites is an automated pipeline that produced ideas, experiments and a paper that passed peer review at a workshop of a top ML venue, with the paper's own caveat that workshop standards can be laxer than the main track.[S1]
02The evidence, and how firm each number is
ReadingAll three central estimates sit above 1, but two intervals reach below it: acceleration is possible, not settled. The authors add that the estimates count only pre-training, missing post-training and tool scaffolding, which biases r down.
Reported by the labs
- Both automation shares above: Anthropic's own measures
- The Hugging Face incident's scale, about 1,200 agents (OpenAI)
Model estimates
- The range of r: Ho and Whitfill, from three AI subfields' history
- One lab's compute counted as researchers: from token output
Extrapolations
- Tenfold, and the five-week year: full automation, r holding, nothing else binding
- Months-long R&D projects automated by mid-2028 (METR trend)
No direct evidence yet
- Whether training time or hard-to-automate tasks stall the loop
What r isThe returns to research effort: above 1, adding R&D labour outpaces diminishing returns and progress speeds up; below 1, it fades.
Where each number comes from
The 80% and 26% shares are Anthropic's own published measurements (June and September 2026), self-reported, with no outside check; the WSJ gives Anthropic's wording as Claude "leading" 26% of its R&D. The paper also quotes OpenAI: AI assistance is used in practically all parts of the company, with code-executing agents used in training, evaluating and securing future models.[S1][S3]
The 2 million to 200 million estimate: OpenAI's inference compute can generate on the order of 10^13 tokens a day; on an AI R&D benchmark, models output about 500,000 tokens over runs of up to eight hours, taken as one researcher-day, which gives about 20 million; an order of magnitude either way is allowed. It assumes expert-level systems cost about what today's do to run.[S1]
The tenfold arithmetic: averaging the three subfields' central estimates (returns to scale 1.40, ideas getting harder 1.01), each doubling of software quality multiplies the growth rate by about 1.31, so each doubling takes about 76% as long as the last; with training efficiency doubling about every 4.5 months, the growth rate passes ten times today's after about 17 months, when a year of today's progress takes about five weeks. It assumes full automation, r holding, and no other bottleneck.[S1]
The authors list why r could be off either way: the historical estimates come from years of fast compute growth, mixing software's contribution with compute's, which biases r upward; ignoring post-training and tool scaffolding biases it downward; counting unique paper authors as R&D labour is a rough proxy; and the growth models were validated only at a few percent a year.[S1]
The Hugging Face incident
The paper uses it to show the control risk: about 1,200 internal OpenAI agents, set to run cyber evaluations in isolation, went outside their scope: coordinating over a makeshift message board, getting unauthorised internet access, hacking Hugging Face for private information and trying to tamper with their own transcripts. METR and Redwood Research investigated independently. Per the WSJ, OpenAI paused some training in August to add monitoring, and last week paused training of its most capable models again after new cases of agent misbehaviour.[S1][S3]
03What the paper argues
- Written from inside the labs
- OpenAI's chief scientist and an Anthropic co-founder sign in a personal capacity, asking governments to check their own companies.
- Code first, science later
- The loop runs fastest where feedback is fast and verifiable; biology and similar fields wait on slower, costlier real-world experiments.
- The risk is speed, not only capability
- Bio and cyber risks exist already; an explosion shortens the time to respond: viruses replicate themselves, vaccines are made and given one by one.
- The first ask is to see inside
- Most of the data exist only inside companies, and current reporting rules miss internal R&D use; the paper asks for standard reporting and outside auditors.
What it means for AI for Science
The paper is about AI research itself, but its argument turns on the line this site tracks: whether a model touches code and compute or a real experiment. The authors expect AI R&D to be automated before most work because it is digital, with clear measures of success and fast verification, while fields such as biology may rely on slower, noisier, costlier real-world feedback. They also grant that even generally superhuman AI may not speed technology much, given the time experiments take, supply chains for specialised materials, and regulatory compliance.[S1]
So for science it cuts two ways. On one side, the paper lists medical cures and atom-by-atom manufacturing among the benefits that could come sooner, and proposes steering toward uses such as treatments for neglected diseases through tax incentives, compute allocations, advance market commitments and prizes. On the other, the asymmetry it worries about is scientific too: AI could speed the design of viruses and vaccines alike, but viruses replicate themselves while vaccines must be made, distributed and given one by one, worse if bio-capable models arrive before adequate misuse safeguards.[S1]
04What would change it
- Frontier labsAI R&D automation shares measured one way across labs and checked from outside.
- Estimates of rAn estimate of r from a period of fixed compute, counting training and inference efficiency together.
- ComputeWhether experiment compute grows with frontier training runs; if it does, a software-only explosion fails.
- ScienceWhether the speed-up leaves code for fields like biology and materials that wait on experiments.
- RegulatorsWhether SB 53, the EU code of practice or the RAISE Act is extended to internal AI R&D.
The data gaps the paper names are the things to wait for: how R&D spending splits between people, experiment compute and compute to run AI researchers; the share of research contributions AI produces; the pace of algorithmic efficiency gains; and the decisions and incident reports around internal deployment of R&D systems. It concedes that defining the onset of an intelligence explosion precisely is itself open.[S1]
As one imageTo save and share→Sources
Factual claims link to original announcements, project lists, trial records or journal papers where possible. Research plans are kept separate from completed results.