AI FOR SCIENCE · INTERVIEWS
A University of Washington professor who builds AI for biomedicine says frontier models have improved through coding and long tasks, not biomedical knowledge; they refuse biomedical work more often, so his group runs much of it on Kimi K3 and Muse Spark; and of designing an experiment a lab can actually run, a model does about 30%.
The record holds the scores and the safety gates as the model makers state them; this episode gives the user's side: which work one academic group hands to which model and why, and how long cells, mice and monkey studies each take, the validation path that sets the pace of AI drug discovery.
Almost all of it is his own experience and judgment, from one group; he does not say how he arrived at the 30%, and gives no rate for the refusals.
His group has moved its internal biomedical work from the Claude models to GPT-6 Astra. Two things make it useful: computer use, which lets the model drive the old programs the field still depends on and finds hard to use, and long runs of 8 to 24 hours, so fast models get the daytime and Astra the overnight pipeline. But he says Astra shows no gain on benchmarks that test biomedical knowledge. On Terminal-Bench-Science's biomedical problems he did worse than the models himself, so he counts dry-lab coding work as close to solved.
He says Astra, Opus and Fable guard biomedical tasks ever more closely and often refuse, which he calls a very serious problem. His group mostly uses Kimi K3 or Muse Spark 1.3 internally; to use Astra they have to explain that the task is not dangerous, for instance that they are building a benchmark. His conclusion is that as capability rises, the experience of actually using the models gets worse.
A model can propose a molecule, but it is poor at the details: the synthesis route, how to store the compound, the dosing schedule in mice, which experts then have to fix. Nor does it know what equipment and materials a given lab has: like asking for a seafood banquet in a kitchen with tomatoes, eggs and potatoes. He calls the missing part 'human use': how an AI directs the technicians who run the experiments.
Cell and mouse studies take a few weeks; a monkey study takes about four months and sometimes runs twice; an AI drug company usually licenses a drug to a large pharma company once the monkey work is done, six to twelve months in all. A model designs a molecule in hours, and by the time the animal work is done the model may be a generation older. So he argues that serious AI drug companies and frontier labs need wet labs of their own, redesigned for AI: cells ready in advance, the molecule added as soon as the prediction arrives.
The breakthroughs he sees are almost proportional to the length of the experiment: enzymes and proteins, where experiments are quick, will iterate fast; ageing and rare diseases will be slow. One experiment run through a whole animal is worth tens to hundreds of thousands of dollars as data. The benchmarks he wants have no top score: if today's drug kills 30% of cancer cells, ask the AI for one that kills 40%, and collect data as it goes; a task such as sorting genes into cancer-causing or not is soon saturated.
The record shows the biology safety gate from the company's side: on 2026-09-17 Anthropic opened a Life Sciences Verification Program that loosens Claude's biology limits for vetted teams, and Opus 5.5 on 22 September was the first model to trigger it.
Wang gives the user's side: Astra, Opus and Fable all often refuse biomedical tasks, so his group mostly uses Kimi K3 and Muse Spark 1.3, and tells Astra it is building a benchmark. He does not say whether his group has applied for verification.
The record's scores for GPT-6 Astra on Terminal-Bench-Science come from the model makers: 64.6% in Anthropic's Opus 5.5 release, 68.1% in OpenAI's GPT-6.1 Sol release.
Wang puts the scores back in the lab: he did worse than the models on those biomedical problems, so he counts dry-lab coding as nearly solved, while saying Astra did not improve on knowledge benchmarks and that the gain came from coding and long tasks.
The record's AI-scientist entries report an agent reaching a result, such as Anthropic's ART, where agents ran a 21.5-hour survey and people did the wet-lab work; none says how much of a model-written protocol a lab can follow as written.
Wang gives an estimate: of designing an experiment, a model does about 30% and experts fix the rest; what is missing is the detail of synthesis, storage and dosing, and designing around the equipment a lab actually has.
If a university group working on biomedical AI moves its work to Kimi K3 and Muse Spark because of refusals, then who a verification program vets, and how fast, decides which tier of model academia works with. It is one group's experience, not a survey.
The gap he names is inside the lab: one lab's equipment, materials and budget, and the steps of synthesis and dosing. Benchmarks scored on predictions do not reach it, so evaluating an 'AI scientist' means asking how much of its protocol a lab can follow as written.
On his account, enzymes and binders, where an experiment returns in days, will show results first, and drugs that wait on monkeys and patients will be slow. That gives a yardstick for reading AI drug-discovery news: first ask how long the experiment behind a result takes.
His group moved internal biomedical work from the Claude models to GPT-6 Astra, uses its computer use to drive old biomedical software, and runs it overnight on 8- to 24-hour tasks.
Current system
His description of his own group; he says they did not compare the models on an objective benchmark.
Astra, Opus and Fable refuse biomedical tasks more and more; the group mostly uses Kimi K3 or Muse Spark 1.3, and tells Astra it is building a benchmark.
Current system
One group's practice, with no refusal rate given; whether it applied to Anthropic's verification program is not said. The two model names are read from a machine transcript that spells them 'keamy k3' and 'muse bark 1.3'.
Astra's gain in biomedicine comes from coding and long tasks; on benchmarks of biomedical knowledge it did not improve.
Guest's judgment
He does not name the benchmark.
Dry-lab biomedical coding is close to solved: on Terminal-Bench-Science's biomedical problems he did worse than the models, though Astra scores only in the 60s.
Guest's judgment
His own attempt; 'the 60s' fits the 64.6% and 68.1% the model makers report in the record.
Of designing a real experiment a model does about 30%, and experts fix the rest; it does not design around the equipment and materials a lab has.
Guest's judgment
His estimate; he does not say how he arrived at it.
In his group, a model's design goes to a collaborator who then designs a wet-lab experiment afresh; the two halves are not joined.
Not yet closed
He says himself that the split is inefficient.
Cell and mouse studies take weeks, a monkey study about four months and sometimes two; AI drug companies often license to large pharma after the monkey work, six to twelve months in all, usually for more than $10M.
Industry context
His account of common industry practice, with no company or source named.
Adaptyv Bio works with Anthropic, takes designs through MCP and starts the experiments, and has just raised $40M.
Industry context
Adaptyv's own post of 2026-08-25 states a $40M Series A led by Highland Europe and says Claude-designed proteins went to its automated lab; the MCP detail was not read there.
Frontier labs that want high-value discoveries will all need wet labs of their own, and OpenAI will likely do the same.
Guest's judgment
Anthropic's wet lab is on the record (2026-09-18); an OpenAI wet lab is not.
John Jumper has joined Isomorphic Labs.
Industry context
This conflicts with the record, which has Jumper leaving Google DeepMind for Anthropic (June 2026).
Breakthroughs are almost proportional to experiment length: fast-experiment fields such as enzymes and proteins iterate first, ageing and rare diseases slowly.
Guest's judgment
His judgment.
Whether a public evaluation has labs run model-written protocols as written and reports how many steps needed changing.
Would show up as A protocol evaluation scored by execution in a lab, not by matching text.First evidence Duke's PEARL (preprint, 2026-10-21, on the timeline) had GPT-5.6 Terra set concentrations and timings and run an enzyme assay on an Opentrons OT-2, in eight scenarios the authors set and scored on their own rubric. It shows a model planning and running experiments on an ordinary liquid handler, but it does not measure how much of an expert's protocol a model can write.Whether refusals ease for vetted academic groups, or groups like his keep running on open-weight models.
Would show up as A survey or paper reporting which models academic biomedical groups use and how often they are refused.Whether the Anthropic and Adaptyv protein design competition publishes wet-lab hit rates for its 5,000-plus designs, problem by problem.
Would show up as Hit rates per problem, which would test whether fast-experiment fields move first.Whether OpenAI builds a wet lab or signs a wet-lab validation partner in its own name.
Would show up as An announcement by OpenAI or by a named partner.Still open After the episode OpenAI acquired Orchestra Bio (2026-10-20, on the timeline), which is software that tracks a drug programme's work, not a lab.Later evidence that tests or limits what was said, and targets whose date has passed.
Wang says models do not design around the equipment a lab has. Two days later Duke's PEARL preprint connected a general model to an ordinary pipetting robot and a webcam and gave it only a goal; it set concentrations and timings and noticed a plate a person had turned 180 degrees. It is one bench and scenarios the authors set, an early counterexample rather than a test of the 30% figure.Timeline entry →
Read 2026.10.25 This is a three-guest roundtable (E256, recorded on 29 September 2026), and this page covers only Sheng Wang's parts on biomedicine (roughly minutes 9 to 12, 25 to 30 and 47 to 78). What was read is a full machine transcript this site made from the episode audio with Whisper (the small model), not checked by a person and with many misheard words; the guest's name and post come from the show notes and his University of Washington page. The episode's YouTube copy was added on 2026-10-26 (same episode; the text still comes from the audio transcript, not YouTube's captions). The other two guests' parts (Zheqing Zhu, founder of Pokee.ai, on agents; Huazhe Xu, a Tsinghua assistant professor who founded a robot company, on robots) were read too and are outside this page. Open the interview ↗The episode on YouTube (E256) ↗