Probing the limitations of multimodal language models for chemistry and materials research

Nawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García, Indrajeet Mandal, Pranav Khetarpal, Hargun Singh Grover, N. M. Anoop Krishnan, Kevin Maik Jablonka

TL;DR

Researchers have introduced MaCBench, a comprehensive new benchmark designed to test how well Vision-Language Models (VLLMs) handle the multimodal reality of chemistry and materials science. The study reveals a sobering "reasoning gap": while top-tier models like Claude 3.5 Sonnet and GPT-4o excel at basic visual perception—like identifying a beaker—they struggle significantly with spatial reasoning, multi-step logic, and integrating information across different formats.

What Problem Does This Paper Address?

Scientific research is inherently multimodal. A chemist doesn't just read text; they interpret NMR spectra, analyze XRD patterns, and navigate complex laboratory setups. While Large Language Models (LLMs) have shown promise in text-based scientific tasks, their "vision-capable" counterparts (VLLMs) remain largely unvetted in specialized domains.

The authors identify a critical gap: existing benchmarks often focus on general visual abilities or text-only scientific reasoning. We don't yet know if these models can truly act as "scientific assistants" capable of bridging the gap between a visual observation (like a crystal structure rendering) and a scientific conclusion (like identifying its space group).

Methods

The authors developed the MaCBench framework, structured around three fundamental pillars of the scientific workflow:

  1. Data Extraction: Parsing tables, plots, and chemical diagrams from literature and patents.
  2. Experiment Execution: Understanding laboratory safety, identifying equipment, and interpreting simulation artifacts like crystal structures.
  3. Data Interpretation: Analyzing experimental results, including spectroscopy (NMR, MS), microscopy (AFM), and diffraction (XRD).
Figure 1: Overview of the MaCBench framework, covering the multimodal chem- istry and materials science research life cycle. The framework evaluates VLLM perfor- mance across three key domains: data extraction (teal), in silico and laboratory experiments (purple), and data interpretation (pink). The benchmark includes diverse tasks spanning tables, plots, organic chemistry diagrams, crystal structures, atomic force microscopy (AFM) imaging, spectroscopy, and materials characterization. Each task requires domain-specific visual understanding and scientific reasoning, from extracting numerical values to analyzing complex experimental setups and interpreting spectroscopic data. We use icons created by Rainy Ting (on svgrepo.com).

To dig deeper than just a leaderboard score, the team conducted extensive ablation studies. They tested how performance changed when the same information was presented as text versus an image, how models handled increasing reasoning steps, and how sensitive they were to specific scientific terminology versus plain language.

Datasets

The study introduces the MaCBench corpus, consisting of 1,155 questions across nine distinct task categories.

  • Sources: Data was manually sourced from patents, generated from scratch (e.g., new lab setup photos), and curated from experimental data.
  • Diversity: Tasks range from identifying hand-drawn molecules to calculating the volume of a unit cell from a 3D rendering.
  • Validation: Every question was reviewed by multiple scientists to ensure accuracy and define acceptable numerical tolerances.
Figure 1: Overview of the MaCBench framework, covering the multimodal chem- istry and materials science research life cycle. The framework evaluates VLLM perfor- mance across three key domains: data extraction (teal), in silico and laboratory experiments (purple), and data interpretation (pink). The benchmark includes diverse tasks spanning tables, plots, organic chemistry diagrams, crystal structures, atomic force microscopy (AFM) imaging, spectroscopy, and materials characterization. Each task requires domain-specific visual understanding and scientific reasoning, from extracting numerical values to analyzing complex experimental setups and interpreting spectroscopic data. We use icons created by Rainy Ting (on svgrepo.com).

Key Insights and Ideas

  • The Perception vs. Reasoning Paradox: Models achieve near-perfect scores in identifying lab equipment (perception) but drop to near-random guessing when asked to identify isomeric relationships or space groups (spatial reasoning).
  • Modality Fragility: Models perform significantly better when scientific data is provided as text (e.g., a list of peak positions) than when it is shown in an image (the actual XRD plot). For XRD peak identification, switching from image to text resulted in a 35% performance boost, suggesting VLLMs haven't yet mastered cross-modal synthesis.
  • Pattern Matching vs. First Principles: The authors found a striking correlation between a model's success on a specific crystal structure task and how often that structure appears in Google search results. This suggests models may be relying on "memorized" patterns from their training data rather than performing genuine scientific reasoning.
  • Terminology Sensitivity: Surprisingly, removing technical jargon sometimes improved performance. Models appear overly sensitive to specific technical vocabularies (like SMILES strings) rather than understanding the underlying chemical concepts.

Key Results

  • Leaderboard: Claude 3.5 Sonnet emerged as the top performer across all three task families, followed by GPT-4o and Gemini 1.5 Pro. Llama 3.2 90B Vision struggled significantly, often performing near the random baseline on data extraction tasks.
  • Reasoning Decay: Performance consistently degraded as tasks required more logical steps. For example, models were good at finding the highest peak in a plot but failed at ranking the relative intensities of multiple peaks.
  • Safety Refusals: GPT-4o frequently refused to answer questions related to lab safety (50% refusal rate) and crystal symmetry, likely due to over-active safety alignment filters.
Figure 3: Performance of frontier VLLMs. a. Accuracy gains compared to random baseline across three core scientific tasks, showing varied performance of Claude 3.5 Sonnet, GPT-4o, Gemini Pro, and Llama 3.2 90B Vision in averaged across all task in the three focus areas of MaCBench: data extraction, experimental understanding, and interpretation tasks. We show the performance as the fraction of correctly answered questions relative to a random baseline. A performance of 0 means that the model is indistinguishable from random guessing. The error bars indicate the standard deviation of the fraction of correctly answered questions over five different runs. b. Radar plot demonstrating the relative model performance across ten specialized scientific domains. Again, we show the fraction of correctly answered questions relative to a random baseline (the plots without the normalization are shown in Figure A.1). We can observe substantial differences in performance across topics.

Limitations and Future Work

The primary limitation identified is the fundamental lack of spatial reasoning in current VLLM architectures. The models "see" the pixels but fail to construct a robust 3D mental model of molecules or crystals.

Future Directions: * Synthetic Training: Developing specialized datasets with explicit 3D spatial relationships to teach models about rotations and perspectives. * Modality Translation: Training models specifically on tasks that require converting between representations (e.g., turning a spectrum image into a data table). * Test-Time Scaling: Leveraging "Chain-of-Thought" reasoning or agentic frameworks to help models handle multi-step logical inferences more reliably.

In short, while VLLMs are promising "assistants" for simple data entry and object recognition, they aren't ready to take the lead in the creative or complex analytical phases of materials research.

Summary updated

Abstract

Recent advancements in artificial intelligence have sparked interest in scientific assistants that could support researchers across the full spectrum of scientific workflows, from literature review to experimental design and data analysis. A key capability for such systems is the ability to process and reason about scientific information in both visual and textual forms - from interpreting spectroscopic data to understanding laboratory setups. Here, we introduce MaCBench, a comprehensive benchmark for evaluating how vision-language models handle real-world chemistry and materials science tasks across three core aspects: data extraction, experimental understanding, and results interpretation. Through a systematic evaluation of leading models, we find that while these systems show promising capabilities in basic perception tasks - achieving near-perfect performance in equipment identification and standardized data extraction - they exhibit fundamental limitations in spatial reasoning, cross-modal information synthesis, and multi-step logical inference. Our insights have important implications beyond chemistry and materials science, suggesting that developing reliable multimodal AI scientific assistants may require advances in curating suitable training data and approaches to training those models.
BibTeX
@inproceedings{alampara2025probing,
  author = {Nawaf Alampara and Mara Schilling-Wilhelmi and Martiño Ríos-García and Indrajeet Mandal and Pranav Khetarpal and Hargun Singh Grover and N. M. Anoop Krishnan and Kevin Maik Jablonka},
  title = {{Probing the limitations of multimodal language models for chemistry and materials research}},
  booktitle = {Nature Computational Science},
  year = {2025},
  doi = {10.1038/s43588-025-00836-3},
}
Figure