Paper & Benchmark· Evidence · Frontier

Why Does AI Misread Molecular Structure Drawings?

OCSR already scores highly on traditional benchmarks, yet real papers test both visual recognition and chemical representation: even a clear drawing can express chemistry that common molecular formats fail to preserve.

Updated 2026.08.229 min read

Real-paper benchmark + technical-report evaluation

Current boundary：validation is still limited to a selected real-paper benchmark and under-covers patents, older scans and complex Markush structures

This work advances：Recovers molecular structures from real scientific images

MODELUnderstand / predict
DECIDEChoose next
INTERACTAct / measure
UPDATEChange next round

Series Context

MinerU.Chem series | From chemistry PDFs to machine-usable reaction data

PDFChemical regionsMolecular structuresReaction relationshipsStructured data

01 | What happened

Real papers expose two distinct OCSR failure modes

The previous brief followed a chemistry PDF through the MinerU.Chem pipeline. This one focuses on its hardest cell: reconstructing a machine-readable molecular graph from a two-dimensional molecular drawing, or Optical Chemical Structure Recognition (OCSR).^[S3]

Accuracy already exceeds 90% on many older benchmarks, which can make the task appear nearly solved. Real papers do not look like clean small molecules on white backgrounds. Scan noise, blurred bonds and crowded layouts are only the first kind of difficulty.^[S1]

The second kind is different. A clear image can still fail because it contains non-standard bonds, unusual valence, radicals, coordination structures, Markush notation, polymer repeat units or a Ph abbreviation. A drawing can be visually difficult, chemically difficult, or both.^[S1]

02 | What Changed

Separating visual recognition from chemical representation

Separating two kinds of difficulty

MolRecBench-Wild's MOSAIC framework separates OCSR difficulty into two dimensions. Visual difficulties include low resolution, scan noise, crowded layout, overlapping text and broken or blurred bond lines. They test whether the model can see reliably.^[S1]

Chemical-semantic difficulties remain even with perfect pixels: unusual valence, radicals, coordination structures, abbreviations, Markush and polymer representations. The question changes from whether the model can see to what representation can express the chemistry it saw.^[S1]

The benchmark contains 5,029 real molecular drawings from 820 recent chemistry papers and 37 MOSAIC difficulty classes: 18 visual and 19 chemical-semantic. At least one difficulty appears in 93.29% of samples, while 42% combine both kinds. That distribution explains why legacy-benchmark scores can be overly optimistic.^[S1]

Why image-to-SMILES can fail

Many vision-language OCSR systems resemble image captioning: an image goes in and a SMILES string comes out. SMILES is highly serialized, so one wrong atom, bond or stereocenter can alter the rest of the sequence. A final string alone also makes it hard to locate the visual error.^[S3]

GTR-VL reconstructs a graph first. It walks the molecular graph, generating atoms and bonds before moving onward. The intermediate graph is inspectable: an error can be localized to an atom or bond and mapped back to the image. That is more useful for database production than knowing only that a whole SMILES string failed to match.^[S3]

The figure says Ph. Why does the answer contain a phenyl ring?

Some ground-truth pipelines expand Ph into a complete phenyl ring. The image contains no drawn six-membered ring, yet the target requires one. The model is being supervised to produce something that is not visually present.^[S3]

GTR-VL calls its principle Faithfully Recognize What You've Seen. If the source says Ph, it is preserved as a superatom and can be expanded during later standardization. The team assembled roughly 1.3 million training examples around this approach. Extraction and normalization should not be collapsed into the first step.^[S3]

Even the evaluation can mark a correct answer wrong

A reconstructed graph may match the ground truth while a different traversal order, abbreviation expansion or canonicalization produces a different SMILES string and fails exact match. Conversely, matching canonicalized SMILES does not prove that abbreviations, 2D coordinates or non-standard semantics were faithfully preserved.^[S1][S3]

MolRecBench-Wild therefore preserves CARBON, graph-level representations and SMILES. As models approach the ceiling of old benchmarks, the model is not the only thing that may need revision. The definition of correctness may need revision too.^[S1]

03 | Key figures

5,029 real drawings redraw the benchmark boundary

5,029

real molecular structures in MolRecBench-Wild v1^[S1]

820

source chemistry papers^[S1]

MOSAIC difficulty categories^[S1]

18 + 19

visual + chemical-semantic difficulties^[S1]

93.29%

contain at least one difficulty^[S1]

42%

contain both visual and chemical difficulty^[S1]

04 | Why it matters

Structure errors propagate through the chemistry data stack

Versions must be kept separate. The original MolRecBench-Wild paper uses 5,029 drawings. The later MinerU.Chem report uses a revised set with 5,024 fully graph-annotated structures, of which only 2,392 can be validly converted to SMILES. The denominators should not be mixed.^[S1][S3]

MolRecBench-Wild is not making an exam harder for its own sake; it moves the exam toward the environment where the model must work. If production data contains Markush, polymers, abbreviations, scanned patents and crowded supporting-information figures while the test contains only clean small molecules, 90% accuracy says little about usability.

OCSR also sits at the entrance to the chemistry-data pipeline. One wrong stereobond can propagate into reaction parsing, databases, property links and model training. The dangerous result is not an invalid SMILES but a valid, plausible molecule that differs from the source.

WHAT CHANGED

AI4S layer: Scientific data / Optical Chemical Structure Recognition
Original bottleneck: Real papers combine visual complexity with chemical-semantic complexity
What changed: MolRecBench-Wild rebuilds evaluation from real papers; GTR-VL recovers a molecular graph before SMILES
Key evidence: Faithfully Recognize What You've Seen
Still unsolved: Long-tail chemistry, complex visual interference and inconsistencies in representation and evaluation

EVIDENCE IN CONTEXT

Editorial status: Frontier
Evidence setting: Retrospective or simulated
How the evidence was produced: Retrospective
Provenance and access: Paper and public dataset
Evidence ceiling: It shows that real-paper images are harder than older benchmarks, but does not establish coverage across eras, patents and chemistry subfields.

05 | Evidence status and boundaries

A more realistic benchmark is not production-wide validation

A real-paper benchmark is closer to production than older datasets, but it is not the whole chemistry literature. The 5,029 drawings come from 820 selected recent papers and cannot represent every publisher, era, scan quality, patent collection or chemistry subfield. MOSAIC labels and complex ground truth also require human judgment.^[S1]

The conclusion is not that we now know the true accuracy of OCSR. It is that older benchmarks likely made the task look easier, and a more production-relevant ruler now exists.

06 | What I Learned

Representation and evaluation are part of the model problem

A benchmark should resemble where the model will workIf future data contains Markush structures and the test set never does, a beautiful leaderboard is not enough.

Scientific representation is not a neutral file formatSMILES, graphs and CARBON preserve different information and change what can be considered correct.

Annotation mismatches can be more fundamental than model sizeIf the image says Ph but ground truth requires a phenyl ring, scale will not automatically repair the supervision mismatch.

Knowing where a result went wrong mattersAn inspectable molecular graph is often better suited to data production than a black-box final string.

Sources

Factual claims link to original announcements, project lists, trial records or journal papers where possible. Research plans are kept separate from completed results.