Demis Hassabis says recursive self-improvement moves fastest where answers can be verified (maths, code, games); today it makes an engineer three to five times as productive with a human still reviewing, and taking the human reviewer out of the loop would be the most dangerous way to reach AGI.
Self-improvement moves fastest where answers can be checked: maths and code
Hassabis says recursive self-improvement works best in domains where an answer can be verified: maths, coding and, first, games. His example is AlphaZero, which used no human data, started from random play against itself, and within a day went from a chess player he could still just about beat to stronger than any chess program. A verifiable answer lets the current best version set new problems and generate almost unlimited synthetic data while knowing whether that data is right; that, he says, is why maths and coding have moved so fast.
After AlphaFold, the next step is predicting toxicity and what a drug does in the body
On medicine, he says protein structure is a critical but small part of drug discovery: to design a drug you need the protein's 3D surface to know where to bind. AlphaFold 2 was finished in 2020, he says, and about four million researchers use it; the next steps are biochemistry and predicting toxicity and ADME (how a drug is absorbed, distributed, metabolised and excreted), which a drug needs before it is safe and effective.
Removing the human reviewer is the danger; he asks for sandboxes, monitoring and outside audits
Today's self-improvement is early, he says: it makes an engineer three, four or five times as productive, with the human still reviewing code and designing the architecture. Taking the human reviewer out and closing the loop would be faster, and in his view the most dangerous way to reach full AGI, ending in an intelligence we may struggle to understand and that may struggle to understand us. What he proposes: give trusted cyber defenders models fine-tuned for defence first, harden sandboxes, monitor agents so none runs for weeks unnoticed, and in time seek formal guarantees, work he urges pure mathematicians unhappy with the state of their field to take up through complexity and game theory. He also argues for an industry standards body with government oversight, third-party auditors and labs red-teaming each other before release, and calls the letter the labs signed better than nothing.
For now, build intelligent tools, not conscious entities
He holds that intelligence and consciousness can come apart, and that for now we should build intelligent tools, not conscious entities: for safety, since a system that can refuse instructions on its own judgment is at odds with one that follows them and can be stopped, doubly so with closed-loop self-improvement; and because society has not yet debated what moral standing such systems would have. The tools could first be used for neuroscience, to find the neural correlates of consciousness, which he expects to take five to ten years. All of this is his judgment and proposal; the conversation shows no new data or result.