AIntibody: 511 antibodies, one set of assays, and which task AI wins or loses

Three tasks, one ruler

AIntibody follows the CASP structure contest: the tasks are set first, submissions are sealed, and the organizers measure every result under one protocol. 29 organizations entered, which the paper groups as 10 academic or nonprofit labs, 8 AI biotechs, 5 small-to-mid biotechs, 5 large pharma companies and 1 big tech company. There is one target, the receptor-binding domain of the SARS-CoV-2 spike protein, chosen deliberately: a well-studied target with rich data, in a challenge the authors describe as primed for success.^[S1]

Task one is in-library optimization: given the sequencing output of the first phase of a yeast-display campaign, write sequences that bind more tightly and remain developable, changing five of the six loops but not the heavy-chain third loop or the framework. No binding data was provided beyond the parental antibody. 25 organizations submitted 165 sequences.^[S1]

Task two is ranking: inside three clusters grouped by their heavy-chain third loop (2,524 sequences in 27F, 3,554 in 28F, 410 in 47F), pick the one antibody in each that binds most tightly and is developable. A few measured affinities came with each cluster (22, 14 and 3 respectively), along with the cluster control, the sequence that appears most often in the sequencing. Task three is out-of-library design: using all 32,059 sequences and 142 measured affinities, design sequences that are not in the dataset, leaving the framework untouched.^[S1]

The measurement layer is the hardest part of this work. Every submission was first measured on a chip (high-throughput surface plasmon resonance), and the tightest binders were re-measured by an equilibrium method in solution (KinExA). Rank correlation between the two solution formats was 0.94 and 0.97, and 0.92 against the chip: absolute values differ between methods, with the solution method generally reporting tighter binding, but the rank order is stable. Developability is a composite of five assays: hydrophobic-interaction chromatography retention, nonspecific binding, self-interaction, melting temperature and aggregation temperature, each scored 0 to 2, with a total of 3 or less counting as a pass.^[S1]

What each task returned

Task one: of 165 submissions, 118 (71.5%) bound, 22 (13.3%) reached single-digit nanomolar, 5 (3.0%) went below 1 nM and 1 (0.6%) below 100 pM; 86.1% passed developability and 63.0% were both binders and developable. The winner, Aureka, reached 95 pM, about 2,000 times tighter than the parental antibody, with a 95% interval overlapping the lab's own best antibody at 113 pM, so the two are statistically indistinguishable. Aureka's sequence sits 19 substitutions from the parent and at least 12 from anything in the experimental dataset, an unusually bold move. In the same task, 8 organizations could not produce a single developable binder.^[S1]

Task two: 10.3%, 13.8% and 9.8% of submissions beat the cluster control in the three clusters. The authors measured the baseline on the same sequences: 39% of randomly picked clones beat that control. For most entrants, using a model to choose was worse than taking the most abundant sequence and picking a few more at random. The one exception was Washington University in St. Louis (50% against 39%), and even that method was ineffective in cluster 27F. No winning algorithm carried across all three clusters.^[S1]

Task three: of 168 submissions, 51 (30.4%) did not bind at all, 24 (14.3%) went below 100 pM, and 53.6% were both binders and developable. By the challenge rules the winner was a 2.9 pM design, tighter than the lab's own best at 9.2 pM. It also failed to elute from the hydrophobic-interaction column, a failure that under ordinary drug-development criteria would probably disqualify it, and it won because the composite score lets high affinity offset one assay failure. The next developable design, at 8.69 pM, was statistically tied with the lab's best.^[S1]

Before

Hit rates were self-reported, each on its own target, assays and selection rules^[S1]

Bottleneck

No prospective, blinded comparison measured under one protocol

What changed

Three tasks, 511 antibodies, one assay suite, with the baseline measured too^[S1]

New bottleneck

Whether any of it holds on an unfamiliar target with sparse data

What remains unsolved

Transfer across tasks, and the groups with published capability that did not enter^[S1]

The most usable thing in this paper is that it splits 'is AI useful for antibodies' into three questions and answers each against a baseline. Buy per task, not per company: a group placing first in one task and far outside the leaders in another was the norm here, not the exception. The paper's cross-task rank table shows that every disclosed group had at least one task where its best submission fell well behind.^[S1]

The makeup of the winners is worth recording too. Only one winner, Aureka, describes itself as an AI company. The others were three academic groups (Scripps, UCSD and Washington University in St. Louis) and a mid-sized biotech, Xencor, which makes no claim to AI expertise and took one cluster and the out-of-library task with two different algorithms. The authors read this as a field where deep pockets are not yet decisive.^[S1]

There is a lesson about benchmark design, and the authors state it themselves: a composite developability score let one failure be offset elsewhere, so a design that would not come off a chromatography column took first place. They say the next round will add hard go/no-go criteria. Scoring rules decide what entrants optimize for, which is not a problem unique to antibodies.^[S1]

This site puts the result on rung 4 of the evidence ladder: every submission was physically made and measured on instruments rather than scored back on existing data. Three further things carry its weight: the tasks were set first, submissions sealed and measurement done by third parties under one protocol; the baseline was measured rather than assumed; and two independent measurement methods agreed on rank order.^[S1]

It is worth being clear about how forgiving this contest was. There is one target, among the best-studied antigens there is, and the data was generous: tasks two and three even supplied 142 measured affinities, which in a real campaign are usually not available until late. The authors write that only task one used the kind of dataset produced under normal conditions. So this round answers whether models can optimize when data is plentiful, not whether they can discover when it is sparse.^[S1]

The authors also flag a weakness in the blinding: it rested on the organizers' integrity rather than on any cryptographic check in the data, and they say the next round will address it. Entering also required disclosing some method detail, which the authors think kept away groups with trade-secret concerns, and several groups with publicly claimed capability did not appear.^[S1]

Read by method class, the winning class differs in every task: task one went to a protein language model paired with three-dimensional structure (Aureka), task two's three clusters went to classical machine learning, a custom transformer and a language model, and task three to a language model (Xencor). The authors read this as language models helping most where design leaves the library, and helping most inside it when combined with structure, with no method winning everywhere.^[S1]

The baseline decides what a result is worth

In task two the AI improvement rates do not look bad on their own (9.8% to 13.8%). Only with the 39% from random picking beside them does the number turn into bad news. A baseline is not decoration on a benchmark; it is the unit the reading comes in.^[S1]

Scoring rules are incentives

A composite developability score let one failure be offset elsewhere, so task three was won by a molecule that would not come off a chromatography column. The authors say hard go/no-go criteria are coming. Any benchmark that compresses several measurements into one number owes the same answer: which failure must never be offset.^[S1]

This reading would strengthen if the next round withholds affinity data and changes to an unfamiliar target while the same models still match the lab on optimization, or if an outside group reproduces the rank order on another target. It would weaken if the winning methods turn out to depend on public data for this one antigen, or if a wider field of entrants substantially rewrites the order.

One note on this site's own record: several antibody-design entries on the timeline report hit rates the groups measured themselves (91 to 100% from Latent Labs, 38 of 80 designs binding for Aureka, 17 of 211 for Anthropic). This round overturns none of them, but it supplies a ruler: a hit rate with no blinding and no baseline should be read for what it is.