TL;DR
ChemCoTBench-V2 is a novel, rule-verifiable diagnostic benchmark designed to evaluate the intermediate reasoning steps of Large Language Models (LLMs) on chemistry tasks, rather than just scoring their final answers. By testing 5,620 samples across 18 tasks, it reveals a persistent gap in frontier models: LLMs frequently output the correct final molecule or product while relying on flawed, chemically impossible intermediate logic.
What Problem Does This Paper Address?
Most current chemistry benchmarks evaluate LLMs purely as question-answering systems, scoring only the final output (such as a SMILES string or a reaction product). However, this outcome-only view hides a critical failure mode: a model can easily arrive at the correct answer by making impossible bond changes, misidentifying scaffolds, or violating basic conservation rules along the way.
Evaluating the actual process of chemical reasoning is notoriously difficult. Human step-level annotation is too expensive to scale, and using an LLM-as-a-judge is unreliable because models can generate fluent but chemically invalid explanations. To solve this, the authors introduce a scalable, low-cost, and auditable framework that uses deterministic chemical rules to verify intermediate reasoning states.
Methods
The authors developed a unified three-layer evaluation protocol that distills natural chemical Chain-of-Thought (CoT) patterns into expert-refined, structured templates:
- Layer 1 (Outcome Correctness): Evaluates only the final answer using standard task-specific metrics (e.g., exact molecular match, MAE, Tanimoto similarity).
- Layer 2 (Template Adherence): Checks if the model successfully follows the requested structural format and maintains internal consistency between fields, without evaluating the underlying chemistry.
- Layer 3 (Step-wise Verifier Correctness): The core innovation. It evaluates the actual chemical validity of intermediate steps using two types of checks:
- Type-I (Intrinsic Symbolic Checks): Computes rule-based predicates directly on the model's state using cheminformatics tools like RDKit (e.g., checking SMILES validity, ring counting, heavy-atom arithmetic, and atom conservation).
- Type-II (Benchmark-State Agreement): Compares closed-answer intermediate states against verified reference traces. For open-ended molecular optimization, Layer 3 uses oracle-computable constraints instead of strict trace matching.
Datasets
The active evaluation set contains 5,620 rule-verifiable samples covering 31 fine-grained tasks grouped into 18 reporting tasks. These tasks span four core chemical families:
- Molecular Understanding: Functional group counting, ring counting, Murcko scaffold extraction, ring-system scaffold judgment, and SMILES equivalence.
- Molecule Editing: Site-specific add, delete, and substitute operations.
- Molecular Optimization: Single- and dual-objective optimization for physicochemical properties and bioactivity targets.
- Reaction Prediction: Product-level prediction, retrosynthesis, template/mechanism reasoning, component recommendation, condition ranking, and yield prediction.
Data was sampled and normalized from public resources including PubChem, ChEMBL, ZINC, Schneider 50K (USPTO), and the Therapeutics Data Commons (TDC).
Key Insights and Ideas
- The "Right for the Wrong Reasons" Phenomenon: Models adopt the requested formal style and templates almost perfectly (Layer 2 scores are frequently $\ge 0.970$), but their chemical reasoning drops sharply in Layer 3. A model can easily tell you that two SMILES strings are equivalent (high Layer 1 accuracy) while failing to maintain a chemically consistent intermediate trace.
- Local vs. Global Tracking: LLMs are reasonably robust when a chemical property can be inferred from local string patterns or substructure cues. However, their performance degrades heavily when they must maintain and update an explicit 2D graph topology (like tracking a ring set or a Murcko scaffold) across multiple steps.
- The Constraint-Coupling Collapse: In molecular optimization, models perform well on single-objective tasks (e.g., improving a physicochemical property). However, when forced to balance dual objectives simultaneously, their joint success rate collapses (dropping to under 10%). They can optimize for one property but fail to keep the global molecular state stable under a second coupled constraint.
Key Results
The authors evaluated 8 frontier LLMs, splitting them into "thinking" (reasoning-oriented) and "no-thinking" models:
- Thinking Models Lead, But Still Struggle: Reasoning-oriented models like Gemini-3.1-Pro and Qwen3.5 Plus generally outperformed standard instruction-following models on Layer 3 metrics. For instance, Gemini-3.1-Pro achieved the highest scores across most molecular understanding and editing tasks.
- The Reasoning Gap in Action: In the SMILES Equivalence task, Claude-Sonnet-4.6 achieved a top-tier Layer 1 accuracy of 96.7%, yet its Layer 3 Type-II all-match score was only 45.3%. This highlights how outcome-based metrics severely overestimate a model's true chemical understanding.
- Scaffolds as Anchors: Prompt ablation experiments on DeepSeek-V3.2 showed that providing explicit intermediate templates and "anchors" significantly improves molecular optimization success rates (jumping from 15.9% to 32.7%), proving that structured scaffolding helps models maintain chemical commitments.
Limitations and Future Work
While ChemCoTBench-V2 provides a powerful diagnostic tool, it is currently limited to 2D molecular and reaction representations. The authors note that extending this process-level verification to more complex domains—such as 3D conformational reasoning, quantum chemistry, laboratory procedure planning, and long-horizon synthesis—remains an open challenge. Future work will focus on defining task-specific states for these broader regimes and using the benchmark's localized failure signals to better train and prompt tool-augmented chemistry assistants.
Summary updated