Why Does AI Misread Molecular Structure Drawings? Paper Drawings Are Harder Than Patent Benchmarks

The previous brief followed a chemistry PDF through the MinerU.Chem pipeline. This one focuses on its hardest cell: reconstructing a machine-readable molecular graph from a two-dimensional molecular drawing, or Optical Chemical Structure Recognition (OCSR).^[S4]

The paper's examples are MolGrapher and MolParser, which both exceed 90% exact-match accuracy on benchmarks such as USPTO and UOB, which can make the task look nearly solved. The authors note that common benchmarks (DECIMER, CLEF, Staker, USPTO, JPO and others) come mostly from patents or generated images, with standardized drawings, clean backgrounds, high resolution and little noise. Drawings in papers are different: scan noise, blurred bonds and crowded layouts are only the first kind of difficulty.^[S1]

The second kind is different. A clear image can still fail because it contains non-standard bonds, unusual valence, radicals, coordination structures, Markush notation, polymer repeat units or a Ph abbreviation. A drawing can be visually difficult, chemically difficult, or both.^[S1]

Separating two kinds of difficulty

MolRecBench-Wild's MOSAIC framework separates OCSR difficulty into two dimensions. Visual difficulties include low resolution, scan noise, crowded layout, overlapping text and broken or blurred bond lines. They test whether the model can see reliably.^[S1]

Chemical-semantic difficulties remain even with perfect pixels: unusual valence, radicals, coordination structures, abbreviations, Markush and polymer representations. The question changes from whether the model can see to what representation can express the chemistry it saw.^[S1]

The drawings come from about 820 CC BY 4.0 papers in seven chemistry journals. The team first screened about 6,548 candidate images, hand-labelled their MOSAIC difficulties, then picked 5,029 so that samples spread evenly across the two-dimensional space of visual versus chemical difficulty counts. Of the 37 classes, 18 are visual and 19 chemical-semantic; among the benchmark's drawings, 93.29% carry at least one difficulty and 42% carry both kinds. Because the set was picked to balance difficulty, those shares describe this benchmark, not how often such drawings occur in the chemistry literature.^[S1]

On these drawings the paper evaluates 18 OCSR-capable models, and every category drops sharply: the best SMILES accuracy is MolScribe's 41.05%, with GTR-Mol-VLM at 40.43%; on the stricter full-graph metric the best is Gemini 2.5 Pro at only 13.04%. For every model, SMILES scores above simplified graph and simplified graph above full graph: how strictly the answer is checked changes the score by itself.^[S1]

Why image-to-SMILES can fail

The GTR-VL paper sorts existing methods into two kinds. ChemVLM, ChemDFM-X, OCSU and similar systems treat OCSR as image captioning: an image goes in and a SMILES string comes out. Graph-parsing methods such as MolGrapher, MolScribe and MolNexTR predict atoms first, then bonds, to assemble a molecular graph. The trouble with writing SMILES directly is that SMILES is highly serialized, so one wrong atom, bond or stereocenter can alter the rest of the sequence, and a final string alone makes it hard to locate the visual error.^[S3]

GTR-VL also outputs a graph, but not in two stages like earlier graph parsers: it walks the molecular graph, interleaving atoms and bonds as it moves onward. The intermediate graph is inspectable: an error can be localized to an atom or bond and mapped back to the image. That is more useful for database production than knowing only that a whole SMILES string failed to match.^[S3][S4]

The figure says Ph. Why does the answer contain a phenyl ring?

The GTR-VL paper notes that MolScribe's and MolNexTR's patent training data come from MOL files whose ground truth expands Ph into a full phenyl ring. That is chemically equivalent, so the expansion is not itself an error; the issue is fidelity to the drawing. The image says "Ph" and draws no six-membered ring, yet the target requires one. When the supervision target does not match what is drawn, the model learns to guess an expansion where the drawing has an abbreviation.^[S3]

GTR-VL calls its principle Faithfully Recognize What You've Seen. If the source says Ph, it is preserved as a superatom and can be expanded during later standardization. The team assembled roughly 1.3 million training examples around this approach. Extraction and normalization should not be collapsed into the first step.^[S3][S4]

Versions must be kept separate. 5,029 is one version of the benchmark, the one in the original MolRecBench-Wild paper; 5,024 is the revised annotated set, all graph-annotated, used in the MinerU.Chem report; 2,392 is the subset of it that converts to valid SMILES and can be scored on SMILES. The rest contain Markush structures, unconventional abbreviations or bonds that break valence rules, and do not convert. The three denominators should not be mixed.^[S1][S4]

MolRecBench-Wild is not making an exam harder for its own sake; it moves the exam toward the environment where the model must work. If production data contains Markush, polymers, abbreviations, scanned patents and crowded supporting-information figures while the test contains only standardized, clean drawings, 90% accuracy says little about usability.

OCSR also sits at the entrance to the chemistry-data pipeline. One wrong stereobond can propagate into reaction parsing, databases, property links and model training. The dangerous result is not an invalid SMILES but a valid, plausible molecule that differs from the source.

Even the evaluation can mark a correct answer wrong

One molecule can be written as many SMILES strings, so three comparisons need keeping apart: exact match on the raw string, match after canonicalization, and graph equivalence. A reconstructed graph may match the ground truth while a different traversal order, abbreviation expansion or canonicalization yields a different raw string that fails exact match; the GTR-VL paper adds that MolScribe's and MolNexTR's evaluations replace abbreviations with an asterisk (*), leaving those parts unscored. MolRecBench-Wild's SMILES track compares canonicalized sequences, and to make that possible it simplifies non-standard bonds in the labels to ordinary ones. Conversely, matching canonicalized SMILES does not prove that abbreviations, 2D coordinates or non-standard semantics were faithfully preserved.^[S1][S3]

Scores by difficulty subset

The MinerU.Chem report splits the revised benchmark by MOSAIC labels into three non-overlapping subsets: A has none of the specified chemical-semantic difficulties and relatively few visual ones (1,987 drawings); B also has no chemical-semantic difficulty but more visual ones (1,976); C contains the specified chemical-semantic difficulties (1,061). The report's own GTR-VL-1.4.13 scores 92.15%, 80.11% and 55.42% graph accuracy on them; the best SMILES comparison system, GPT-5.6-Sol, scores 44.44%, 28.14% and 18.57%. Each kind of difficulty lowers the score, the chemical-semantic kind more. Note that GTR-VL's training data include paper drawings hand-annotated under the MolRecBench-Wild protocol, and the report does not say whether they were kept apart from the test set.^[S4]

MolRecBench-Wild therefore preserves CARBON, graph-level representations and SMILES. As models approach the ceiling of old benchmarks such as USPTO, the model is not the only thing that may need revision. The definition of correctness may need revision too.^[S1]

A more realistic benchmark is not production-wide validation

A benchmark drawn from papers is closer to the difficulties papers actually contain than patent benchmarks are, but it is not the whole chemistry literature, still less a sample of how often each kind of drawing occurs. The 5,029 drawings come from about 820 recent CC BY papers in seven journals, picked to balance difficulty, and cannot represent every publisher, era, scan quality, patent collection or chemistry subfield. MOSAIC labels and complex ground truth come from 47 annotators with chemistry backgrounds, and the paper reports no inter-annotator agreement.^[S1]

The conclusion is not that we now know the true accuracy of OCSR. It is that patent benchmarks such as USPTO likely made the task look easier, and a ruler closer to how drawings look in papers now exists.

A benchmark should resemble where the model will work

If future data contains Markush structures and the test set never does, a beautiful leaderboard is not enough.

Scientific representation is not a neutral file format

SMILES, graphs and CARBON preserve different information and change what can be considered correct.

Annotation mismatches can be more fundamental than model size

If the image says Ph but ground truth requires a phenyl ring, scale will not automatically repair the supervision mismatch.

Knowing where a result went wrong matters

An inspectable molecular graph is often better suited to data production than a black-box final string.