How Does AI Read a Chemical Reaction Diagram? From Recognized Molecules to Reaction Relationships

From recognized molecules to structured reaction routes

The first two briefs covered the chemistry-PDF pipeline and reconstruction of a molecular graph from a two-dimensional drawing. Suppose every molecule is now correctly localized and recognized. A new problem begins.^[S4][S5]

Are the two molecules on the left reactants? Is the one on the right necessarily the product? Which step owns H2O2, Amberlite IR 120H, solvent or temperature written near an arrow? How do five or six arrows, branches and loops form a route?

Reaction Diagram Parsing must understand molecular objects, text, spatial layout and their relationships before producing a database-ready reaction record.^[S1]

Real reaction diagrams require production-like data

MolDet-33k contains 219,721 manually annotated molecular bounding boxes across 12,209 page-level and 21,155 figure/table-level images after cleaning. The updated RxnCaption data covers 15,128 reaction-parsing training images.^[S1][S2]

Scale is not the only change. A left-to-right single-step reaction is not comparable to a total-synthesis scheme with rows, branches, loops and tree-like dependencies. The data deliberately covers different layouts. As throughout this series, production distribution often matters more than benchmark size.^[S1]

Do not ask one large model to do two different jobs

A direct approach asks one VLM to locate every molecule and infer every role. Scene understanding and precise coordinate regression are different tasks; a shifted box can corrupt both identity and relationship.^[S1]

RxnCaption uses an engineering split. MolYOLO first locates molecules and overlays indexed boxes. The VLM no longer predicts coordinates; it describes relationships among numbered objects: 1 and 2 are reactants, 3 is product, and nearby text is a condition.^[S1][S2]

The authors call this BIVP, BBox and Index as Visual Prompt. It looks like simply drawing boxes first, but represents a mature scientific-AI architecture: specialist perception followed by foundation-model semantic reasoning.^[S1]

If the paper already says 1a and 2b, why renumber them?

BIVP solves coordinates by modifying the image. RxnID's IdtVP instead uses native compound identifiers as visual-semantic handles. If the paper says 'compound 1a was converted to 2b,' the model can say 1a → 2b rather than remembering 'the third molecule in a blue box.'^[S3]

1a and 2b began as a publishing convention but become natural grounding tokens for a multimodal model. In the paper's zero-shot evaluation, Gemini 3 Pro with IdtVP reports 71.70% Hybrid Match F1 and 88.11% Soft Match F1 on RxnScribe-test, and 86.5% Soft Match F1 on scanned historical literature. Those numbers span two metrics and two datasets, so they cannot be ranked against each other.^[S3]

These numbers do not mean 86.5% of reaction diagrams are completely correct. Soft Match checks only the molecule boxes (IoU > 0.5), ignoring text and folding conditions into reactants; Hybrid Match is stricter, also requiring condition text within a normalized edit distance of 0.2. Both tolerate some differences, and neither is the rate of database records requiring no correction.^[S3]

Many real molecules have no identifier

Identifiers are useful but not free ground truth. An intermediate may be unnumbered, the same 7 may reappear in another figure, an identifier may sit far from its structure, or a compound may be named only in prose.^[S3]

RxnID adds Mid-Mapper: molecules with native identifiers are linked directly where possible, while unnumbered molecules receive virtual identifiers. At this point the task moves from reading one image toward understanding a document, and cross-page reference resolution becomes a new bottleneck.^[S3]

The more important result is an emerging scientific-AI architecture: specialist perception, foundation-model reasoning and structured verification. MolYOLO localizes, a VLM interprets reaction relationships, compound identifiers provide grounding and reaction-level metrics check structured output.^[S1][S3]

This is closer to a real data system than sending the full image to a larger multimodal model. Connected to MinerU.Chem, the goal is no longer merely understanding a diagram; it is making decades of schemes in papers and supporting information usable by retrosynthesis, reaction prediction and chemistry knowledge bases.

How much native identifiers add over drawn-on indices

RxnID's zero-shot comparison (Table 1) puts three visual-prompt strategies on the same Gemini 3 Pro. On RxnScribe-test, BROS (the raw image, with the model regressing coordinates) reaches only 3.90% Hybrid Match F1 and fails to produce valid graph structures; BIVP's drawn-on indexed boxes reach 45.27%; IdtVP's native identifiers reach 71.70%. On the harder RxnCaption-15k-test the three score 4.92%, 47.40% and 58.30%. The same IdtVP on smaller models gives 16.69% for Qwen2.5-VL-7B and 44.71% for Intern-S1. The authors' reading is that high-precision zero-shot parsing needs effective visual guidance and strong reasoning in the foundation model together.^[S3]

Zero-shot prompting and trained models are separate results

The paper runs two kinds of experiment. One is prompting: general VLMs (Gemini 3 Pro, GPT-4o, Qwen2.5-VL-7B and others) with no task fine-tuning, only a different visual prompt (Table 1 and the lower half of Table 3). The other is training: a short SFT cold start on BIVP- or IdtVP-format data, then Re³-DAPO reinforcement learning, producing RxnCaption and RxnID (the upper half of Table 3). Table 3 lists both, so compare within one dataset and one metric: on Soft Match, zero-shot Gemini 3 Pro + IdtVP is highest on all three test sets (88.1% / 78.2% / 86.5%); on the stricter Hybrid Match it is highest only on the scanned ScannedRxn set (59.3%, against 56.3% for RxnID+RL), and trails on RxnScribe-test (71.7%, against 75.9% for RxnCaption+RL and 75.0% for RxnID+RL) and on RxnCaption-15k-test (58.3%, against 64.4% for RxnID+RL).^[S3]

Why use verifiable rewards?

Reaction parsing often ends in JSON, but a reaction is not a string with one valid order. Swapping parallel reactants or independent reactions does not change the chemistry. Next-token loss can still penalize a semantically correct answer for a different serialization order.^[S3]

RxnID's Re³-DAPO raises reward to the reaction-set level, judging an output closer to Hybrid/Soft Match rather than demanding token-for-token identity. The interesting point is not merely the use of RL, but aligning the training objective with the scientific object. It does help in distribution: on RxnCaption-15k-test, RxnID's Hybrid F1 rises from 61.2% to 64.4% and the BIVP model's from 59.8% to 64.1%. On scanned papers, though, the BIVP model falls from 51.0% to 45.1% after RL, which the authors attribute to degenerate token repetition. RL was run once (about 70 hours on eight H200s), with no multi-run variance analysis.^[S3]

Verifiable rewards have boundaries. A cropped arrow, illegible condition or broken cross-page identifier can still yield beautiful JSON. Reward can test consistency with annotation rules; it cannot prove that a reaction occurred, its yield is reliable or its conditions reproduce.

These benchmarks show meaningful improvement in reaction-diagram parsing, but do not answer the document-level question: how many reaction records from a full paper can enter a database without human correction? Errors from detection, OCSR, OCR, image cropping and identifier matching all propagate here.^[S1][S3][S4]

Hybrid and Soft Match are task-specific evaluations, not scientific validity. Reaction parsing can extract A + B → C under condition X; it cannot validate success, yield, mechanism or reproducibility. It solves information extraction, not chemical validation.

A foundation model should not do every job

Giving precise localization to a detector and relationship reasoning to a VLM is a more mature system design.

Scientific notation can become an AI interface

The identifiers 1a and 2b began as publishing conventions but have become useful multimodal grounding interfaces.

Training objectives should match scientific objects

A reaction is a graph or set, not a string that must appear in one order.

The final measure should be end-to-end data yield

Users do not need a 98% detector, 93% OCSR and high parser F1 in isolation. They need to know how many high-confidence, traceable records a PDF yields without correction and how long the rest take to repair.

Putting the three briefs back into one chain

A general parser finds chemistry regions, OCSR turns pixels into molecular graphs, identifiers connect structures to context, and reaction parsing reconstructs reactants, products, conditions and relationships. No single 93% or 98% score represents the system. The question is whether the pipeline can reliably turn literature into high-quality, traceable, machine-usable scientific data.