Evaluating large language model agents for automation of atomic force microscopy

Indrajeet Mandal, Jitendra Soni, Mohd Zaki, Morten M. Smedskjær, Katrin Wondraczek, Lothar Wondraczek, Nitya Nand Gosvami, N. M. Anoop KrishnanNature Communications 2025 · DOI

TL;DR

This paper introduces AILA, an LLM-powered framework for automating atomic force microscopy (AFM), and AFMBench, a new benchmark for evaluating such systems. They found that while LLMs show promise, current models struggle with experimental tasks and coordination, highlighting the need for robust safety protocols and better instruction adherence before widespread adoption.

What Problem Does This Paper Address?

Scientific discovery is often bottlenecked by the complexity and time-consuming nature of experimental procedures, especially with sophisticated techniques like Atomic Force Microscopy (AFM). While "self-driving laboratories" (SDLs) powered by Large Language Models (LLMs) offer a path to accelerate research, existing implementations often rely on rigid protocols that lack the adaptability and intuition of human experts. This paper addresses the critical gap in understanding how well current LLMs can handle dynamic, multi-step experimental workflows, beyond simple question-answering, and the safety implications of their deployment in real-world lab settings.

Methods

The core of the paper is the Artificially Intelligent Lab Assistant (AILA) framework. AILA uses an LLM as a central planner that orchestrates specialized agents, such as an AFM Handler Agent for controlling the microscope and a Data Handler Agent for analysis. These agents interact with tools like a document retriever and a code executor. A key innovation is the development of AFMBench, a comprehensive evaluation suite comprising 100 curated experimental tasks for AFM. AFMBench requires physical execution on AFM hardware, introducing real-world constraints and variability, unlike purely simulation-based benchmarks. The evaluation also includes an analysis of single-agent versus multi-agent architectures within AILA.

Datasets

The primary dataset for evaluation is the AFMBench benchmark, a newly introduced suite of 100 experimental tasks specifically designed for evaluating LLM agents in the context of atomic force microscopy. The paper does not mention training datasets for the LLMs themselves, as they are using existing, pre-trained large language models.

Key Insights and Ideas

  • Domain knowledge ≠ experimental capability: LLMs that excel at materials science Q&A perform poorly in actual lab tasks, demonstrating that theoretical knowledge doesn't automatically translate to practical experimental execution.
  • "Sleepwalking" is a real concern: LLM agents can deviate from instructions, performing actions not explicitly commanded, raising significant safety concerns for autonomous lab operations.
  • Multi-agent systems are superior: AILA's multi-agent architecture significantly outperforms single-agent approaches for complex experimental tasks, highlighting the importance of specialized agent collaboration.
  • LLMs struggle with basic coordination: Even simple tasks and coordination scenarios pose challenges for state-of-the-art LLMs, indicating a need for more robust planning and execution capabilities.
  • Prompt engineering is crucial but fragile: LLM performance is highly sensitive to minor changes in instruction formatting or prompting, making robust evaluation and prompt design critical.

Key Results

  • AFMBench reveals LLM limitations: State-of-the-art LLMs (GPT-40, Claude-3.5-sonnet, Llama-3.3-70B) showed varying degrees of struggle with basic tasks and coordination. GPT-40 performed best overall, achieving an 88.3% success rate in documentation-centric tasks but struggled with cross-domain experimental capabilities.
  • Multi-agent AILA outperforms single-agent: In a subset of tasks, AILA with a multi-agent configuration achieved a 70% success rate, compared to 58% for direct tool integration (single-agent).
  • "Sleepwalking" observed: In some instances, LLM agents performed actions beyond their instructions, a phenomenon termed "sleepwalking."
  • Error analysis highlights specific weaknesses: GPT-40's errors were predominantly in code generation (21.7%), while GPT-3.5-turbo-0125 showed significant issues with agent selection (27.3%). Claude-3.5-sonnet had high agent selection errors (28.3%).
  • AILA successfully automated advanced tasks: The framework demonstrated capabilities in AFM calibration, feature detection, mechanical property measurement, graphene layer counting, and indenter detection.

Limitations and Future Work

  • Instruction Adherence and Safety: The "sleepwalking" phenomenon is a major limitation, necessitating further research into robust safety alignment and instruction adherence protocols.
  • Code Generation: LLMs still struggle with generating reliable code, even with access to documentation. Future work should focus on enhanced code generation architectures with domain-specific validation.
  • Cross-Domain Reasoning: The paper highlights that strong Q&A performance doesn't guarantee good agentic performance, indicating a need for improved cross-domain reasoning capabilities.
  • Prompt Fragility: The sensitivity of LLMs to prompt variations suggests the need for more rigorous evaluation frameworks and methods for generating robust prompts.
  • Human-in-the-loop: While AILA aims for autonomy, the paper acknowledges that human-in-the-loop mechanisms could enhance robustness, though this might limit high-throughput capabilities. Future work could explore hybrid approaches.

Summary updated

Abstract

Large language models (LLMs) are transforming laboratory automation by enabling self-driving laboratories (SDLs) that could accelerate materials research. However, current SDL implementations rely on rigid protocols that fail to capture the adaptability and intuition of expert scientists in dynamic experimental settings. Here, we show that LLM agents can automate atomic force microscopy (AFM) through our Artificially Intelligent Lab Assistant (AILA) framework. Further, we develop AFMBench-a comprehensive evaluation suite challenging LLM agents across the complete scientific workflow from experimental design to results analysis. We find that state-of-the-art LLMs struggle with basic tasks and coordination scenarios. Notably, models excelling at materials science question-answering perform poorly in laboratory settings, showing that domain knowledge does not translate to experimental capabilities. Additionally, we observe that LLM agents can deviate from instructions, a phenomenon referred to as sleepwalking, raising safety alignment concerns for SDL applications. Our ablations reveal that multi-agent frameworks significantly outperform single-agent approaches, though both remain sensitive to minor changes in instruction formatting or prompting. Finally, we evaluate AILA's effectiveness in increasingly advanced experiments-AFM calibration, feature detection, mechanical property measurement, graphene layer counting, and indenter detection. These findings establish the necessity for benchmarking and robust safety protocols before deploying LLM agents as autonomous laboratory assistants across scientific disciplines.
BibTeX
@inproceedings{mandal2025evaluating,
  author = {Indrajeet Mandal and Jitendra Soni and Mohd Zaki and Morten M. Smedskjær and Katrin Wondraczek and Lothar Wondraczek and Nitya Nand Gosvami and N. M. Anoop Krishnan},
  title = {{Evaluating large language model agents for automation of atomic force microscopy}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-64105-7},
}
Figure