Poisoned Evidence: Syntax Sensitivity in Molecular Retrieval-Augmented Generation

When a language model answers questions about molecules, it usually reads the molecule as a text string (SMILES) rather than a graph. The same molecule can be written as dozens of equally valid strings — starting the traversal at a different atom, spelling out hydrogens, writing aromatic rings as alternating single and double bonds — and chemical language models are notoriously brittle to these identity-preserving rewrites: with explicit hydrogens added, retrieval accuracy collapses below 5% and solubility-prediction error on ESOL jumps from an RMSE of 0.87 to 7.93, an order of magnitude, as reported by both Measuring Chemical LLM robustness to molecular representations: a SMILES variation-based framework and Lost in Translation: Chemical Language Models and the Misunderstanding of Molecule Structures. Retrieval-augmented generation (RAG) — letting the model look up similar molecules with known labels before answering — is sold as the fix for brittle parametric knowledge, and MolRAG shows it can lift molecular property prediction by 1.1% to 45.7% over direct prediction. But nobody has asked the obvious cross-question: does the SMILES brittleness travel through the retrieval loop? A retrieved neighbor written in a different syntax than the query is textbook "poisoned evidence" — retrieval succeeded, yet the context the model actually reads may be unintelligible to it.

I propose to (a) build the first controlled measurement of how identity-preserving string rewrites corrupt each stage of a molecular RAG pipeline — retrieval, context, and generation — and (b) repair it with evidence rendered in a modality that is invariant to atom-traversal order: 2D structural depictions of the retrieved neighbors, read by a general-purpose vision-language model, alongside invariant textual descriptors. The multimodal reasoning gap supplies both the motivation and the mechanism here: vision-language models can recognize structural features in images but fail to chain them into deduction, which is exactly why the retrieved pictures need to be paired with the same step-by-step contrastive scaffolding (target image versus neighbor image plus known label) that MolRAG's similarity-aware chain-of-thought provides in text.

Core Claim

The claim has a negative part and a positive part. Negatively: molecular RAG is not syntax-invariant, and its failure point is not where practitioners assume. The retrieval stage — when it operates on molecular fingerprints computed by RDKit rather than raw text embeddings — is already robust to traversal-order rewrites, because the parser canonicalizes internally. The damage happens in the context window: when the target or the retrieved neighbors appear as non-canonical, kekulized, or explicit-hydrogen SMILES, the generator's in-context reasoning collapses, reproducing the "canonical crutch" that MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs documents for non-retrieval settings, where high benchmark scores reflect memorized canonical syntax rather than graph understanding. Positively: presenting retrieved neighbors as rendered 2D structure images — a modality with no traversal order, no hydrogen spelling, no ring-numbering convention — restores invariance at the context stage, provided the image evidence is scaffolded with contrastive reasoning templates rather than presented raw.

Formally, let $f$ be the full pipeline (query molecule, retriever, evidence, generator), $q$ the query molecule's string, $D$ the retrieved evidence, and $\pi$ an identity-preserving rewrite from the AMORE family — canonicalization, explicit hydrogen addition, kekulization, cycle renumbering, or random traversal. We say the pipeline is syntax-invariant iff

$$f(\pi(q), \pi(D)) = f(q, D) \quad \text{for every identity-preserving } \pi,$$

meaning the predicted property (or caption) is unchanged, not merely the retrieved set. The proposal is a benchmark that measures violations of this condition stage-by-stage, and a method that reduces them.

Research Program

Phase A: the poisoned-evidence benchmark. I will extend the MolRAG setup — fingerprint retrieval over labeled molecule databases, chain-of-thought generation with a Llama-3-class backbone — across ten MoleculeNet tasks (regression: ESOL, FreeSolv, Lipo; classification: BACE, HIV, Tox21, and the rest of MolRAG's suite), and cross it with the AMORE perturbation suite. Three conditions matter: perturb only the query string, perturb only the retrieved neighbors' strings (simulating a database of patents or experimental readouts that stores non-canonical forms), and the mismatch condition — canonical query, perturbed evidence — which isolates pure context poisoning from query-comprehension failure. Metrics: recall@k of the retrieved neighbor set against the canonical-form run (a control that should be near-perfect for fingerprint retrieval, confirming the damage is downstream), prediction flip rate, and error inflation (RMSE ratio perturbed over canonical). A fourth measurement imports the MolecularIQ "indexing gap": ask the model to localize which atoms carry a retrieved neighbor's shared substructure, since spatial localization accuracy drops 5–30% when models must index atom positions rather than count features.

Phase B: invariant-evidence generation. The repair replaces each retrieved neighbor's SMILES string in the prompt with (i) an RDKit-rendered 2D depiction of the molecule and (ii) an invariant text descriptor: known label, molecular weight, functional group inventory. The target molecule is likewise rendered. The generator is a general-purpose vision-language model — a deliberate choice, since MolecularIQ found large generalists (GPT-OSS 120B, Qwen-3 235B) consistently beat chemistry-fine-tuned specialists, whose fine-tuning eroded instruction-following and general reasoning. The reasoning scaffold is MolRAG's similarity-aware chain-of-thought restated visually: "the target picture shares motif X with neighbor picture Y, whose measured solubility is Z, therefore…". As a text-only fallback arm, I will serialize neighbors in the LAUGHS representation of LAUGHS: An LLM-compatible Molecular String Representation, whose moiety-tree grammar aligns with LLM tokenizers (0.999 alignment score) and whose localized-editing exact match of 91.4% dwarfs SMILES' 69.9% — evidence that if any string form can survive in-context, it is that one.

Phase C: captioning under stress. To test generality beyond property regression, the same perturbation matrix and the same image-evidence repair apply to molecule captioning on ChEBI-20, where Chemical Language Models Have Problems with Chemistry reports the reference damage: MolT5-base METEOR of 0.583 on original strings, 0.329 with explicit hydrogens, and — critically — 0.450 after canonicalization, showing that even the "obvious fix" degrades generation. Retrieval-augmented captioning (retrieve neighbors with their ChEBI descriptions) has never been stress-tested this way; no reported result found on Lacuna for retrieval-augmented ChEBI-20 captioning under SMILES perturbation.

This design also resolves, empirically, a direct contradiction in the literature: DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models found that retrieving in-context demonstrations by question text beat retrieving on image-plus-text features, because image embeddings injected noise into the retrieval space. In chemistry the polarity should invert: the text modality is the corrupted one, and the image modality is the stable anchor. DCoT's finding is a constraint on my design — it warns against image-based retrieval — which is why my retriever stays fingerprint-based and only the context presentation becomes visual.

Baselines

The naive reference point is direct prediction: the same backbone LLM reads the target as canonical SMILES with no retrieval, exactly the condition MolRAG improves upon, and we will run it across the full perturbation matrix so that its fragility is measured on our setup rather than assumed. For scale of expected damage, the reference numbers in the literature are the Text+Chem T5 ESOL results reported by both AMORE and Lost in Translation: RMSE 0.87 on canonical SMILES versus 7.93 with explicit hydrogens.

The strongest competitor is MolRAG (MolRAG: Unlocking the Power of Large Language Models for Molecular Property Prediction), which retrieves structural analogues via Morgan-fingerprint Dice similarity and feeds neighbor SMILES plus known labels into similarity-aware chain-of-thought; it reports gains of 1.1% to 45.7% over direct prediction across ten MoleculeNet datasets and matches supervised graph neural networks on BACE and ToxCast. It is the strongest competitor because it is the only existing system that combines molecular retrieval with in-context analogical reasoning. Fairness conditions: same backbone model, same fingerprint retriever and top-k, and an equalized context budget — image evidence is matched to the token count of the SMILES evidence it replaces by adjusting the number of retrieved neighbors, and every method gets the same prompt-tuning budget on a held-out validation split. Beating it means two things simultaneously: matching its canonical-condition performance within noise, and showing a substantially smaller error-inflation ratio and prediction flip rate under perturbed-query, perturbed-evidence, and mismatch conditions.

Two further comparisons isolate ingredients. The preprocessing baseline canonicalizes every string with RDKit's MolToSmiles before prompting (the strongest version of the "just clean the strings" objection). The key ablation keeps the fingerprint retrieval and the invariant text descriptors but removes the rendered images — if text-only invariant descriptors match the image arm, the visual component is doing no work and the claim reduces to "better strings," which the descriptors-only arm would then own.

Why not just…?

Why not just canonicalize all inputs with RDKit? This is the most serious objection and the preprocessing baseline exists precisely to test it. Two published results argue it falls short. First, canonicalization itself degrades generation: MolT5-base captioning dropped from METEOR 0.583 on original strings to 0.450 after canonicalization in the ChEBI-20 case study, with qualitative errors like misclassifying a substituted ring as a cycloalkane — canonical strings are not the strings models were trained on either. Second, MolecularIQ shows canonical syntax is exactly what models memorize, so a canonicalizer launders the symptom while preserving the crutch. Third, a string canonicalizer cannot process the visual and spectral inputs that real chemistry workflows contain, as CRAG-MoW argues for NMR-integrated search. Still, if the preprocessing baseline matches my method on the full perturbation matrix, the honest conclusion is that for text-only pipelines the canonicalizer suffices; the image arm would then need to win on the visual-input conditions to justify itself.

Why not just use a larger model with longer reasoning? The file contains three independent negative results. MATH-Vision reports the best model (GPT-4o) at 30.39% accuracy against human 68.82%, with 42.2% of GPT-4V errors being pure reasoning errors after successful perception. MMAI Gym for Science states plainly that increasing model size or reasoning tokens does not yield the required scientific gains, and notes failure to generalize across representations such as SMILES to IUPAC. And in MolecularIQ, failed chain-of-thought responses were longer than successful ones — more reasoning tokens amplify confusion rather than repair it. I will still run the largest available backbone as a scaling check within our benchmark, because the objection deserves a number, not just a citation.

Why not just use BM25 with high-quality text extraction? Retrieval or Representation? shows BM25 with better OCR gains up to +31.1 Top-5 points on figure-heavy pages, and I take that result seriously — it is why my retriever is deliberately simple and fingerprint-based. But the same paper states its own boundary: for tasks requiring genuine spatial and structural interpretation, text retrieval hits a ceiling, and GeoSQA shows that translating diagrams into text does not rescue multi-step reasoning. Molecular topology is the extreme case: flattening a graph into a string is the very mechanism that causes the sensitivity being studied, per AMORE and LAUGHS.

Why not just fine-tune a chemistry specialist? MolecularIQ found generalists consistently beating chemistry-fine-tuned models, with naïve domain fine-tuning degrading instruction-following; and the fine-tuned specialists (ChemBERTa, MolT5, nach0) were precisely the models that hit the hydrogen wall in AMORE. The counterpoint is Bolek, which raised mean ROC/PR AUC from 0.55 to 0.76 over its 4B base by injecting a Morgan-fingerprint token with targeted training — proof that structured adaptation can work where naive fine-tuning fails. My proposal is a training-free attempt to get Bolek-style structural grounding through the context window instead of a projector; if the image-evidence arm cannot approach Bolek's invariance at its task, that is evidence the learned-token route is genuinely necessary, and I will say so.

Decisive Checks and Closing Experiments

Three results would settle the proposal. First, the control: fingerprint retrieval recall@k should stay essentially flat across all five perturbation types; if it does not, the story relocates to the retrieval stage and the design changes. Second, the poisoning effect: under the mismatch condition (canonical query, perturbed-evidence SMILES), MolRAG-style text evidence should show material error inflation and flip rates on ESOL/FreeSolv/Lipo — my prediction, anchored to the published 0.87-to-7.93 RMSE swing, is that explicit-hydrogen and kekulized neighbor strings will be the worst offenders while cycle renumbering will be mildest. Third, the repair: the image-evidence arm should hold error within roughly 10% of its canonical-condition performance across the full matrix, while the SMILES-evidence MolRAG arm degrades; and the descriptors-only ablation tells us whether the pictures matter or merely the removal of fragile syntax. A secondary check on ChEBI-20 captioning asks whether METEOR under explicit-hydrogen perturbation stays near the 0.583 canonical reference rather than the 0.329 hydrogen reference. If the poisoning effect turns out to be small — if modern generalist LLMs already read perturbed SMILES gracefully in context — that is itself a publishable negative result that revises the AMORE picture from embeddings to open-loop prompting, and the benchmark would stand as the instrument that showed it.