Agentic Mixture-of-Workflows for Multi-Modal Chemical Search

Tiffany J. Callahan, Nathaniel H. Park, Sara Capponi

Abstract

The vast and complex materials design space demands innovative strategies to integrate multidisciplinary scientific knowledge and optimize materials discovery. While large language models (LLMs) have demonstrated promising reasoning and automation capabilities across various domains, their application in materials science remains limited due to a lack of benchmarking standards and practical implementation frameworks. To address these challenges, we introduce Mixture-of-Workflows for Self-Corrective Retrieval-Augmented Generation (CRAG-MoW) - a novel paradigm that orchestrates multiple agentic workflows employing distinct CRAG strategies using open-source LLMs. Unlike prior approaches, CRAG-MoW synthesizes diverse outputs through an orchestration agent, enabling direct evaluation of multiple LLMs across the same problem domain. We benchmark CRAG-MoWs across small molecules, polymers, and chemical reactions, as well as multi-modal nuclear magnetic resonance (NMR) spectral retrieval. Our results demonstrate that CRAG-MoWs achieve performance comparable to GPT-4o while being preferred more frequently in comparative evaluations, highlighting the advantage of structured retrieval and multi-agent synthesis. By revealing performance variations across data types, CRAG-MoW provides a scalable, interpretable, and benchmark-driven approach to optimizing AI architectures for materials discovery. These insights are pivotal in addressing fundamental gaps in benchmarking LLMs and autonomous AI agents for scientific applications.

Introduction

Materials science is currently facing a "needle in a haystack" problem. The design space for new materials is effectively infinite, and discovering a new polymer or catalyst requires synthesizing knowledge across text, chemical structures (represented as SMILES strings), and experimental data like NMR spectra. While Large Language Models (LLMs) like GPT-4 are powerful, they often struggle with the precision required for chemistry, frequently "hallucinating" or lacking a standardized way to be benchmarked in this specific scientific domain. This paper addresses the need for a scalable, interpretable, and multi-modal framework that can handle the specialized language of chemistry while minimizing errors.

Methodology: The CRAG-MoW Framework

The authors introduce the Mixture-of-Workflows (MoW) architecture. Instead of asking one model for an answer, the system uses a "team" of open-source AI agents to search and analyze complex chemical data.

High-level architecture of the CRAG-MoW system, illustrating the orchestration of multiple generators and the document fusion process.

The system consists of two main components:

  1. Generators: These are individual workflows powered by different open-source LLMs (such as Llama 3.1 or Mistral). Each Generator performs Self-Corrective RAG (CRAG).
  2. Aggregators: An orchestration agent receives the outputs from all the Generators. It uses a technique called Reciprocal Rank Fusion (RRF) to re-rank the retrieved documents and synthesizes a final, high-quality report for the user.

Self-Corrective RAG Workflow

Standard RAG (retrieving data and feeding it to an LLM) is often insufficient for chemistry. The "Self-Corrective" loop—where the AI explicitly checks for hallucinations and completeness—is vital for maintaining scientific accuracy. As shown in the workflow below, the agent doesn't just retrieve data; it evaluates the relevance of what it found, checks its own response for hallucinations, and rewrites the user's query if the initial search fails.

The detailed Self-Corrective RAG workflow and the multi-agent supervisor structure used to handle specialized chemical domains like polymers and NMR characterization.

Multi-Modal Integration

To handle the "multi-modal" aspect, the team used MoLFormer to embed chemical structures and OpenClip to embed NMR spectral images into a unified vector database (Milvus), allowing the AI to "see" both structures and graphs.

Experimental Setup and Datasets

The study utilized a massive library of chemical data:

  • 750,000 structures: Divided equally (250k each) among small molecules, polymers, and chemical reactions.
  • NMR Spectra: A collection of 2,259 $^{13}$C and $^1$H NMR spectra sourced from historical experiments and the NMRShiftDB2 database.
  • Benchmark Questions: A set of expert-curated questions designed to test the system's ability to find structural analogues and characterize spectra.

The "Generators" were powered by a variety of open-source LLMs, including Llama 3.1, Mistral, and Phi-3.5, while GPT-4o served as the primary baseline.

Key Results and Insights

The researchers used an "LLM-Judge" (GPT-4o-mini) to evaluate the responses on a 10-point scale and through pairwise comparisons:

  • Comparable Performance: CRAG-MoW achieved an average score of 7.12, very close to GPT-4o’s 7.59.
  • The Preference Edge: In head-to-head comparisons, responses from the CRAG-MoW system were preferred 8.77% of the time, compared to only 5.89% for GPT-4o. This suggests that the "Wisdom of the Crowd" approach—synthesizing diverse perspectives—effectively filters out the errors of individual models.
  • No Universal Winner: The researchers found that different LLMs have different "personalities" in science. Some are better at understanding polymers, while others excel at chemical reactions.
  • Domain Difficulty: The system struggled most with chemical reactions, which required significantly more search iterations and query rewrites than small molecules.

Conclusion and Future Work

CRAG-MoW provides a scalable, interpretable, and benchmark-driven approach to optimizing AI architectures for materials discovery. While successful, the authors acknowledge several hurdles for future work, including scaling the dataset further, involving PhD-level chemists for validation, and exploring Graph Neural Networks (GNNs) to better represent molecular topology. The ultimate goal is a system that dynamically assigns specific LLMs to tasks based on query complexity, further optimizing the "mixture" of workflows.

Summary updated

Abstract

The vast and complex materials design space demands innovative strategies to integrate multidisciplinary scientific knowledge and optimize materials discovery. While large language models (LLMs) have demonstrated promising reasoning and automation capabilities across various domains, their application in materials science remains limited due to a lack of benchmarking standards and practical implementation frameworks. To address these challenges, we introduce Mixture-of-Workflows for Self-Corrective Retrieval-Augmented Generation (CRAG-MoW) - a novel paradigm that orchestrates multiple agentic workflows employing distinct CRAG strategies using open-source LLMs. Unlike prior approaches, CRAG-MoW synthesizes diverse outputs through an orchestration agent, enabling direct evaluation of multiple LLMs across the same problem domain. We benchmark CRAG-MoWs across small molecules, polymers, and chemical reactions, as well as multi-modal nuclear magnetic resonance (NMR) spectral retrieval. Our results demonstrate that CRAG-MoWs achieve performance comparable to GPT-4o while being preferred more frequently in comparative evaluations, highlighting the advantage of structured retrieval and multi-agent synthesis. By revealing performance variations across data types, CRAG-MoW provides a scalable, interpretable, and benchmark-driven approach to optimizing AI architectures for materials discovery. These insights are pivotal in addressing fundamental gaps in benchmarking LLMs and autonomous AI agents for scientific applications.
BibTeX
@misc{callahan2025agentic,
  author = {Tiffany J. Callahan and Nathaniel H. Park and Sara Capponi},
  title = {{Agentic Mixture-of-Workflows for Multi-Modal Chemical Search}},
  year = {2025},
  eprint = {2502.19629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.19629},
}
Figure