From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models

Hongyu Guo, Hao Li, He Cao, Gongbo Zhang, Li YuanarXiv 2026 · arXiv

TL;DR

ChemCoTBench-V2 is a novel, rule-verifiable diagnostic benchmark designed to evaluate the intermediate reasoning steps of Large Language Models (LLMs) on chemistry tasks, rather than just scoring their final answers. By testing 5,620 samples across 18 tasks, it reveals a persistent gap in frontier models: LLMs frequently output the correct final molecule or product while relying on flawed, chemically impossible intermediate logic.

What Problem Does This Paper Address?

Most current chemistry benchmarks evaluate LLMs purely as question-answering systems, scoring only the final output (such as a SMILES string or a reaction product). However, this outcome-only view hides a critical failure mode: a model can easily arrive at the correct answer by making impossible bond changes, misidentifying scaffolds, or violating basic conservation rules along the way.

Evaluating the actual process of chemical reasoning is notoriously difficult. Human step-level annotation is too expensive to scale, and using an LLM-as-a-judge is unreliable because models can generate fluent but chemically invalid explanations. To solve this, the authors introduce a scalable, low-cost, and auditable framework that uses deterministic chemical rules to verify intermediate reasoning states.

Figure 1: CHEMCOTBENCH-V2 evaluates structured, verifier-addressable chemical reasoning traces beyond final answers with three signals: Layer 1 outcome correctness, Layer 2 template adherence, and Layer 3 step-wise validity under deterministic task-specific checks.

Methods

The authors developed a unified three-layer evaluation protocol that distills natural chemical Chain-of-Thought (CoT) patterns into expert-refined, structured templates:

  1. Layer 1 (Outcome Correctness): Evaluates only the final answer using standard task-specific metrics (e.g., exact molecular match, MAE, Tanimoto similarity).
  2. Layer 2 (Template Adherence): Checks if the model successfully follows the requested structural format and maintains internal consistency between fields, without evaluating the underlying chemistry.
  3. Layer 3 (Step-wise Verifier Correctness): The core innovation. It evaluates the actual chemical validity of intermediate steps using two types of checks:
  4. Type-I (Intrinsic Symbolic Checks): Computes rule-based predicates directly on the model's state using cheminformatics tools like RDKit (e.g., checking SMILES validity, ring counting, heavy-atom arithmetic, and atom conservation).
  5. Type-II (Benchmark-State Agreement): Compares closed-answer intermediate states against verified reference traces. For open-ended molecular optimization, Layer 3 uses oracle-computable constraints instead of strict trace matching.
Figure 2: Unified framework for reference construction and evaluation.

Datasets

The active evaluation set contains 5,620 rule-verifiable samples covering 31 fine-grained tasks grouped into 18 reporting tasks. These tasks span four core chemical families:

  • Molecular Understanding: Functional group counting, ring counting, Murcko scaffold extraction, ring-system scaffold judgment, and SMILES equivalence.
  • Molecule Editing: Site-specific add, delete, and substitute operations.
  • Molecular Optimization: Single- and dual-objective optimization for physicochemical properties and bioactivity targets.
  • Reaction Prediction: Product-level prediction, retrosynthesis, template/mechanism reasoning, component recommendation, condition ranking, and yield prediction.

Data was sampled and normalized from public resources including PubChem, ChEMBL, ZINC, Schneider 50K (USPTO), and the Therapeutics Data Commons (TDC).

Key Insights and Ideas

  • The "Right for the Wrong Reasons" Phenomenon: Models adopt the requested formal style and templates almost perfectly (Layer 2 scores are frequently $\ge 0.970$), but their chemical reasoning drops sharply in Layer 3. A model can easily tell you that two SMILES strings are equivalent (high Layer 1 accuracy) while failing to maintain a chemically consistent intermediate trace.
  • Local vs. Global Tracking: LLMs are reasonably robust when a chemical property can be inferred from local string patterns or substructure cues. However, their performance degrades heavily when they must maintain and update an explicit 2D graph topology (like tracking a ring set or a Murcko scaffold) across multiple steps.
  • The Constraint-Coupling Collapse: In molecular optimization, models perform well on single-objective tasks (e.g., improving a physicochemical property). However, when forced to balance dual objectives simultaneously, their joint success rate collapses (dropping to under 10%). They can optimize for one property but fail to keep the global molecular state stable under a second coupled constraint.
Figure 3: Sample-level diagnostic case from the Qwen3.5 Plus forward-reaction evaluation. The figure shows the prompt constraint, a shortened (zk, rk, fk) trace, and the three-layer diagnosis. Although the model predicts the correct product, its Step-2 reaction-type commitment disagrees with the benchmark-defined RXN_TYPE state, which the verifier localizes as the failure.

Key Results

The authors evaluated 8 frontier LLMs, splitting them into "thinking" (reasoning-oriented) and "no-thinking" models:

  • Thinking Models Lead, But Still Struggle: Reasoning-oriented models like Gemini-3.1-Pro and Qwen3.5 Plus generally outperformed standard instruction-following models on Layer 3 metrics. For instance, Gemini-3.1-Pro achieved the highest scores across most molecular understanding and editing tasks.
  • The Reasoning Gap in Action: In the SMILES Equivalence task, Claude-Sonnet-4.6 achieved a top-tier Layer 1 accuracy of 96.7%, yet its Layer 3 Type-II all-match score was only 45.3%. This highlights how outcome-based metrics severely overestimate a model's true chemical understanding.
  • Scaffolds as Anchors: Prompt ablation experiments on DeepSeek-V3.2 showed that providing explicit intermediate templates and "anchors" significantly improves molecular optimization success rates (jumping from 15.9% to 32.7%), proving that structured scaffolding helps models maintain chemical commitments.

Limitations and Future Work

While ChemCoTBench-V2 provides a powerful diagnostic tool, it is currently limited to 2D molecular and reaction representations. The authors note that extending this process-level verification to more complex domains—such as 3D conformational reasoning, quantum chemistry, laboratory procedure planning, and long-horizon synthesis—remains an open challenge. Future work will focus on defining task-specific states for these broader regimes and using the benchmark's localized failure signals to better train and prompt tool-augmented chemistry assistants.

Summary updated

Abstract

Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.
BibTeX
@misc{guo2026from,
  author = {Hongyu Guo and Hao Li and He Cao and Gongbo Zhang and Li Yuan},
  title = {{From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models}},
  year = {2026},
  eprint = {2606.03660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.03660},
}
Figure