Paper Brief · Molecular Representation· Evidence · Frontier

Information Can Be Lost Before a Molecule Reaches the Transformer

Coverage gaps can remain hidden on conventional drug benchmarks, then grow rapidly when rare elements, isotopes and complex stereochemical information combine. Smirk closes this gap with 165 base tokens.

Updated 2026.09.1013 min read

Peer-reviewed paper + public benchmarks + same-team follow-up preprint

Current boundary：Complete OpenSMILES syntax coverage does not cover 3D conformations or physical environments.

This work advances：Preserves rare elements, isotopes, charge and geometry as molecular strings enter a model

MODELUnderstand / predict
DECIDEChoose next
INTERACTAct / measure
UPDATEChange next round

Update (September 10, 2026)Smirk is used in the same team's 1.8B-parameter MIST, showing that it can enter a large-scale training pipeline. MIST remains a preprint and does not isolate the tokenizer's independent contribution to performance.^[S3]

01 | What happened

Closed vocabularies silently erase rare chemistry

Strong performance on MoleculeNet does not mean a molecular model's tokenizer preserved all input information.

Many chemistry models use atom-wise tokenization, applying regular expressions to split SMILES into atoms and chemical symbols. This usually works for common organic molecules. Blind spots arise mainly in rare combinations of low-frequency elements, isotopes, charge, explicit hydrogen and complex stereochemical information. When the vocabulary lacks a combination, the model may receive only [UNK]. Training does not fail, but the original chemical information is gone—for example in [Au], [Ru], [U], [13CH3], [C@H] and [NH3+].^[S1]

A molecular foundation model first turns a SMILES string into tokens. Many chemistry models use atom-wise tokenization. Common atoms and symbols are handled directly, while an atom carrying isotope, chirality, charge, hydrogen count or class information is enclosed in brackets and treated as one indivisible token.^[S1]

When a closed vocabulary encounters a valid combination it does not know, the program often emits [UNK] rather than failing. Gold, ruthenium, uranium, isotope labels and low-frequency combinations of chirality with charge or coordination geometry can collapse into the same placeholder. The input still looks structurally valid after scientific information has disappeared.^[S1]

The team evaluated 35 tokenizers, including 20 chemistry-specific tokenizers and general NLP tokenizers such as GPT-4o, LLaMA and Gemma, measuring coverage and information loss across REALSpace, MoleculeNet and tmQM.^[S1]

**Molecular tokenizer coverage comparison from the Smirk paper**
Paper Fig. 1: a closed atom vocabulary collapses unseen valid combinations into [UNK], while Smirk continues decomposing bracket atoms into OpenSMILES glyphs. Image from the paper. [S1]

02 | What Changed

Decomposing bracket atoms into 165 composable glyph tokens

Why bracket atoms create a combinatorial vocabulary problem

An OpenSMILES bracket atom can combine isotope, element, chirality, hydrogen count, charge and class. The theoretical grammar permits more than 28 trillion combinations. That is not 28 trillion stable real-world molecules, but it makes exhaustive coverage impossible for a closed vocabulary of a few hundred or thousand atom tokens.^[S1]

[isotope][element][chirality][hydrogen count][charge][class]

Smirk does not memorize every complete bracket atom. It decomposes one into glyphs defined by OpenSMILES. For example, [C@@H] becomes [, C, @@, H and ]. The current paper and repository use a 165-token base vocabulary from which any valid OpenSMILES expression can be constructed.^[S1][S4]

**Coverage metrics for molecular tokenizers across REALSpace, tmQM and MoleculeNet**
Paper Fig. 2: fertility, imbalance, normalized entropy and [UNK] frequency expose coverage differences across tokenizers and datasets. Wadell et al., arXiv:2409.15370 (v3). [S2]

Compressing sequences without recreating the blind spot

Fine-grained decomposition lengthens sequences, increasing Transformer attention cost. Smirk-GPE therefore learns merges over chemical-glyph token IDs. Common combinations compress, while rare combinations fall back to base glyphs without reintroducing [UNK].^[S1]

Smirk-GPE was trained on roughly 262 million molecules. Although the target vocabulary was 50,000, usable merges were exhausted at about 2,300 tokens. It is an engineering trade-off among completeness, sequence efficiency and semantic boundaries.^[S1]

Why MoleculeNet can hide the failure

Common @ / @@ chirality in drug-like molecules is not a universal tokenizer failure. Risk concentrates in combinations of rare elements, isotopes, charge, explicit hydrogen and complex geometry. MoleculeNet mostly reflects conventional medicinal chemistry, so aggregate leaderboards can look normal. tmQM contains more transition-metal complexes and exposes the coverage problem much more strongly.^[S1]

The paper measures silent loss with [UNK] frequency and KL divergence, then pretrains and fine-tunes 18 RoBERTa-style encoders for extrinsic validation. N-gram rankings correlate with downstream rankings at roughly 0.667–0.8 Spearman: useful as a low-cost filter, not a replacement for full training and task evaluation.^[S1]

**N-gram cross-entropy and information loss for molecular tokenizers across datasets**
Paper Fig. 3: unknown tokens sharply increase information loss for several tokenizers on tmQM, while Smirk retains fallback coverage through base glyphs. Wadell et al., arXiv:2409.15370 (v3). [S2]

MIST scales Smirk to 1.8B parameters without isolating causality

The later MIST preprint uses Smirk in 28M- and 1.8B-parameter encoders trained on roughly 245 million and 2 billion molecules, then evaluates them through fine-tuning on more than 400 structure–property tasks. This shows that Smirk can operate inside a much larger training pipeline.^[S3]

MIST changes model scale, training data, compute and fine-tuning setup at the same time, so its gains cannot all be attributed to the tokenizer. It demonstrates scale compatibility and adaptability. Its results across chemical regimes come from task fine-tuning, not zero-shot generalization.^[S3]

WHAT CHANGED

AI4S layer: Representation / Molecular tokenization
Original bottleneck: Closed atom-wise vocabularies were not systematically audited for rare OpenSMILES combinations
What changed: The journal version audits 35 tokenizers and provides a constructive 165-token scheme covering valid OpenSMILES
Key evidence: [UNK] rates, KL loss, 18 encoders and later scaling in MIST
Still unsolved: Independent replication, isolated tokenizer gains and 3D or environmental information beyond SMILES

03 | Key figures

Coverage failures grow with chemical diversity

>28T

theoretical bracket-atom grammar combinations—not real molecules^[S1]

165

Smirk base tokens covering valid OpenSMILES^[S1][S4]

35 / 20

all / chemistry-specific tokenizers in the journal version^[S1]

18.9% / ≈50%

[UNK] share emitted by SPE and APE on MoleculeNet / tmQM^[S1]

0.1 → 40.3

MoLFormer information loss from unknown tokens, MoleculeNet to tmQM (nats/molecule)^[S1]

54%

tmQM molecules that could not convert to the evaluated SELFIES representation because of enhanced stereochemistry^[S1]

RoBERTa-style encoders pretrained and fine-tuned from scratch^[S1]

0.667–0.8

Spearman correlation between n-gram and downstream rankings^[S1]

28M → 1.8B

parameter range of MIST models using Smirk^[S3]

arXiv v3 reports 34 / 19; the peer-reviewed version updates the count to 35 / 20.

04 | Why it matters

Model capability starts with what the input preserves

[UNK] is dangerous because it is silent. Training continues, loss declines and aggregate benchmarks can remain strong after scientifically different structures have collapsed into one symbol. The risky output is not an explicit refusal but a fluent prediction made without knowing that information was removed.^[S1]

The risk is uneven. It concentrates more heavily in transition-metal catalysts, isotopes and radiochemistry, complex coordination structures and some battery materials—the regimes molecular foundation models encounter as they move beyond standard drug benchmarks into materials and energy.^[S1]

The same AI4S problem appears earlier in the input chain. Reaction-diagram parsing can lose objects and relationships when figures become structured data; Smirk audits the next representation loss as SMILES enters a Transformer. Both show that model capability is often bounded first by what the input pipeline lets it see. Continue with:^[S1]

Smirk preserves information already present in the string. It cannot restore conformational ensembles, solvent environments, intermolecular interactions or dynamics that SMILES never encoded. Input fidelity must be audited layer by layer; 100% OpenSMILES coverage is not complete visibility into chemical space.How Does AI Read an Entire Chemical Reaction Route? →

The paper also identifies a benchmark limitation: even the relatively diverse tmQM does not cover the full periodic table, all isotopes, charged species or rare structures such as quadruple bonds. If a research direction falls into these blind spots, a public leaderboard may say little about whether the model can read that chemical regime.^[S1]

05 | Evidence status and boundaries

Syntax coverage, downstream gain and physical completeness are different claims

The mathematical coverage gap is well establishedA few thousand indivisible atom tokens cannot enumerate more than 28 trillion theoretical combinations. Smirk covers valid OpenSMILES by composing a finite glyph vocabulary.^[S1]

Public benchmark measurements are inspectable but same-teamCoverage, [UNK] and KL divergence are measured on public datasets including REALSpace, MoleculeNet and tmQM, with open code. Independent replication beyond the originating team has not yet been established.^[S1][S4]

Eighteen Transformer models provide necessary extrinsic validationSmirk is broadly comparable to atom-wise tokenization on MoleculeNet. Its value is clearer in distributions such as tmQM with rarer, more complex chemistry. Conventional drug-like tasks may show little visible gain.^[S1]

Performance gains are conditionalSmirk is broadly comparable to atom-wise tokenization on MoleculeNet. Its value is clearer in distributions such as tmQM with rarer, more complex chemistry. Conventional drug-like tasks may show little visible gain.^[S1]

MIST establishes scale compatibility, not isolated tokenizer causalityThe 28M–1.8B models change scale, data, compute and fine-tuning together. Cross-regime results come from fine-tuning, not zero-shot transfer, and cannot attribute every gain to Smirk.^[S3]

Peer review strengthens provenance, not physical evidence maturitySmirk remains retrospective computational and public-benchmark evidence, without wet-lab or physical closed-loop validation. MIST remains a preprint.^[S1][S3]

The SELFIES comparison reveals another boundaryIn tmQM, 54% of molecules could not be converted to the evaluated SELFIES representation because of enhanced stereochemistry. String validity, grammar coverage and physical information completeness are three different questions.^[S1]

EVIDENCE IN CONTEXT

Editorial status: Frontier
Evidence setting: Retrospective computational validation
How the evidence was produced: Public benchmarks · Pretraining and downstream fine-tuning
Provenance and access: Peer reviewed · Public code · Same-team follow-up preprint
Evidence ceiling: It establishes lossless tokenization of valid OpenSMILES with a finite base vocabulary and shows same-team use at scale; it does not isolate performance gains, provide independent replication or cover all physical chemical information.

06 | What remains unsolved

Four questions remain after complete OpenSMILES coverage

Independent replication is still missing.The coverage audit, information-loss measurements and large-model application all come from the same core author group; systematic replication by an independent team has not yet appeared.

The tokenizer's independent contribution has not been isolated.MIST changes model scale, training data, compute budget and fine-tuning setup together. It shows that Smirk can scale, not that Smirk is the main cause of model performance.

Existing benchmarks remain incomplete.tmQM includes more transition metals and complex stereocenters than MoleculeNet, but still does not systematically cover all elements, isotopes, charged species and rare bond types.

OpenSMILES completeness is not physical completeness.Smirk can preserve information already encoded in the string, but cannot restore 3D conformations, environments, intermolecular interactions or dynamics that SMILES does not express.

Sources

This brief is based primarily on the peer-reviewed paper, its corresponding arXiv version, the MIST preprint and the official Smirk repository.