TL;DR
ARMOR is an agentic framework that significantly improves chemical reaction feasibility prediction by intelligently orchestrating a diverse suite of AI tools. By explicitly modeling the specific strengths (utilities) of different models and using a memory-augmented reasoning process to resolve tool conflicts, ARMOR achieves a state-of-the-art accuracy of 91.62%, outperforming both individual models and standard ensemble methods.
What Problem Does This Paper Address?
Predicting whether a chemical reaction will actually work (feasibility) is a cornerstone of computational chemistry and automated synthesis. While recent advances in AI have produced various tools for this—ranging from BERT-based classifiers and molecular fingerprint models to Large Language Models (LLMs)—no single tool is a "silver bullet."
A model might excel at predicting reactions involving simple carbon chains but fail on complex aromatic rings. Conversely, an LLM might have great contextual reasoning but suffer from "hallucinations" or biases toward certain reaction classes. The core challenge is tool orchestration: how do we effectively leverage the complementary strengths of multiple models when they inevitably disagree? Existing methods like simple majority voting or heuristic "mixture-of-experts" often fail to capture the nuanced conditions under which a specific tool is truly reliable.
Methods
The authors propose ARMOR (Agentic framework for adaptive utility-aware Multi-tool Reasoning and cOnflict resolution), which treats the prediction task as a multi-step agentic process. The framework consists of three main pillars:
- Tool Hierarchy Construction: Tools are organized into two levels. The first level ($\mathcal{T}^{(1)}$) contains top-performing "generalist" tools for initial decision-making. If these tools agree, the process stops. If not, the "specialist" tools in the second level ($\mathcal{T}^{(2)}$) are brought in.
- Utility-aware Tool Prioritization: Instead of just looking at raw accuracy, ARMOR uses an LLM to extract descriptive "patterns" for each tool (e.g., "Tool A is highly accurate when a new C-N bond is formed in an aromatic ring"). These patterns are refined and consolidated into a utility library. During inference, ARMOR selects the tools whose patterns best match the specific reaction at hand.
- Tool Conflict Resolution: When selected tools produce conflicting predictions, ARMOR doesn't just flip a coin. It uses a Tool Conflict Memory—a database of historical cases where tools disagreed and a "ground truth" rationale was established. It retrieves similar past conflicts to provide few-shot demonstrations, allowing an LLM agent to reason through the current conflict and make a final, informed prediction.
Datasets
The framework was evaluated using the FREA dataset, a benchmark for chemical reaction feasibility.
- Source: Derived from the U.S. Patent & Trademark Office (USPTO).
- Composition: Contains real-world feasible reactions and systematically generated "negative" (infeasible) reactions validated by experts.
- Scale: 12,000 reactions total, split into 6,000 for validation (used to build the tool hierarchy and memory) and 6,000 for testing.
- Tool Set: 13 diverse tools, including 3 classifiers (like BERT and DRFP), 2 generative models (Chemformer, Molecular Transformer), and 8 LLM-based reasoners.
Key Insights and Ideas
- Explicit Utility > Implicit Routing: Traditional Mixture-of-Experts (MoE) models often use "black-box" gating. ARMOR’s use of human-readable chemical patterns (like "double bond formation") makes the tool selection process interpretable and more robust.
- The Value of Specialists: The authors found that some tools with mediocre overall accuracy were nearly 100% accurate on very specific reaction types. ARMOR’s hierarchy allows these specialists to shine exactly when needed.
- Conflict as an Opportunity: The most significant performance gains occurred in "Conflict" cases—reactions where initial tools disagreed. By focusing on resolving these specific discrepancies through memory-augmented reasoning, ARMOR pushed past the limitations of simple ensembling.
- Balanced Performance: Many LLMs are biased toward predicting reactions as "infeasible." ARMOR achieves a very balanced performance across both feasible and infeasible classes, which is critical for practical laboratory use.
Key Results
ARMOR consistently outperformed all baselines, including the strongest individual models and sophisticated dynamic selection methods.
- Overall Accuracy: ARMOR reached 91.62%, surpassing the best single tool (BERT at 87.90%) and the strongest baseline (HarderMoE at 89.50%).
- Balanced Metrics: It achieved an MCC (Matthews Correlation Coefficient) of 0.8323, significantly higher than the next best method (0.7919), indicating superior handling of both positive and negative classes.
- Ablation Success: As shown in Figure 2, removing any single component (Conflict Resolution, Utility Modeling, or Hierarchy) led to a clear drop in performance, proving that the "agentic" orchestration is the key driver of success.
Limitations and Future Work
- Data Dependency: ARMOR currently requires a labeled validation set to characterize tool utilities and build its conflict memory. In real-world deployment, such high-quality labeled data might be scarce.
- Static vs. Dynamic: The current utility patterns are static once extracted. The authors suggest that online adaptation—where the model continues to learn and refine tool utilities during deployment—is a promising next step.
- Generalization: While tested on feasibility, the framework is theoretically generalizable to other "AI for Science" tasks like retrosynthesis or molecular property prediction, which the authors plan to explore.
Summary updated