AtomSD: Procedural Atom-Guided Self-Distillation for LLM Agents

Bohan Yu, Pengfei Cao, Kai Tian, Chen Han, Shi-Yang Li, Chenxi Zhou, Junlin Liu, Cao Liu, Ke Zeng, Jun Zhao, Kang LiuarXiv 2026 · arXiv

TL;DR

AtomSD introduces a modular way for LLM agents to learn "how to act" by breaking down successful experiences into fine-grained Procedural Atoms (Intent, When, Do, and Avoid). By continually extracting these atoms from its own successes and distilling them back into its policy, the agent learns to compose complex behaviors that generalize far better than traditional trajectory-based or skill-bundling methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) are great at "knowing that" (declarative knowledge) but often struggle with "knowing how" (procedural knowledge). An LLM might know the chemical properties of reagents but fail to execute a multi-step lab experiment in the correct order.

Current methods to fix this usually involve:

  1. Full Trajectories: Showing the model a complete successful run. These are often too specific to one scenario and hard to reuse.
  2. Bundled Skills: Grouping actions into "skills." These can be too rigid; a new task might only need half of a pre-defined skill.

The authors argue that we need a more modular, "atomic" unit of procedural knowledge that can be flexibly recombined across different tasks and contexts.

Methods

The researchers propose AtomSD (Atom-guided Self-Distillation), which centers on two main innovations:

1. The Procedural Atom

Instead of a long list of actions, knowledge is represented as an Atom consisting of four parts:

  • Intent: What is the goal?
  • When: Under what conditions is this applicable?
  • Do: What is the specific sequence of operations?
  • Avoid: What are the common failure modes or "trap" actions to steer clear of?
Figure 1: Procedural Atoms enable fine-grained reuse and composition of procedural knowledge.

2. The Self-Distillation Loop

AtomSD uses a "student-teacher" framework to internalize these atoms:

  • Online Abstraction: The student agent attempts tasks. When it succeeds, it acts as its own "knowledge extractor," turning the successful trajectory into a set of Procedural Atoms.
  • Atom-Guided Supervision: A teacher model (an Exponential Moving Average of the student) is given these extracted atoms as context. It then provides token-level guidance to the student on how to solve the task.
  • Joint Optimization: The student is updated using a combination of environment rewards (via GRPO) and a distillation loss that encourages it to match the teacher's atom-guided logic. Over time, the student "internalizes" the atoms, meaning it can perform the procedures even when the explicit atom guidance is removed.
Figure 2: Overview of AtomSD. Procedural Atoms are continually abstracted from the student’s own successful trajectories and distilled back into the evolving policy through Atom-guided self-distillation.

Datasets

The framework was evaluated across four diverse benchmarks requiring complex planning:

  • ALFWorld: Household planning and execution in text environments.
  • ScienceWorld: Multi-step scientific experiments (e.g., thermodynamics, biology).
  • WebShop: Navigating a web interface to find products under constraints.
  • StableToolBench: Multi-turn tool use and API calling.

Key Insights and Ideas

  • The Power of "Avoid": One of the most significant findings is that explicitly representing what not to do is crucial. Removing the "Avoid" component from the atoms led to a massive performance drop (up to 14 points).
  • Compositionality is King: Because atoms are fine-grained, the model can learn an atom in one task (e.g., "how to heat an object") and apply it in a completely different task (e.g., "making dough").
  • Efficiency: Atoms are much more compact than full trajectories. On average, an atom uses only 0.15k tokens, whereas a full trajectory might take 4.97k. This makes them highly efficient for providing context during training.
  • Co-Evolution: As the student gets better, it extracts higher-quality atoms, which in turn makes the distillation process more effective. This creates a virtuous cycle of improvement.

Key Results

AtomSD consistently outperformed strong baselines, including standard Reinforcement Learning (GRPO) and other distillation methods (OPSD, Skill-SD).

  • Performance Gains: AtomSD exceeded the state-of-the-art baseline by up to 9.31 points in average performance across all benchmarks.
  • Scaling Up: Using a Qwen3.5-9B backbone, AtomSD-9B actually outperformed DeepSeek-V4-Pro and came within 5.4 points of GPT-5.5 (a much larger, API-based model).
  • Generalization: The model showed strong performance on "unseen" tasks in ScienceWorld and ALFWorld, proving that it isn't just memorizing trajectories but learning reusable procedural logic.
Figure 3: Ablation of Atom components on Qwen3.5-4B and Qwen3.5-9B. Left: Final benchmark performance. Right: Training rewards after removing Intent, Avoid, or both, alongside the Atom-only setting.

Limitations and Future Work

  • The "Cold Start" Problem: AtomSD relies on the student being able to find some successful trajectories to start the extraction process. If a task is so hard that the model never succeeds initially, there are no atoms to extract.
  • Extraction Quality: The process depends on the LLM's ability to accurately "summarize" its own actions into the Atom format. If the extraction prompt is poorly handled, the resulting guidance could be noisy.
  • Future Directions: The authors suggest exploring multi-agent settings where different agents could share a global "Atom Library," or applying this to even more complex, long-horizon reasoning tasks.

Summary updated

Abstract

Large language models (LLMs) possess broad factual knowledge and strong reasoning capabilities, but solving complex tasks also requires procedural knowledge about how to act. Existing approaches often represent procedural knowledge through trajectories or bundled skills, limiting flexible reuse across interaction histories. This paper introduces Procedural Atoms, fine-grained and independently applicable units that represent Intent, When, Do, and Avoid guidance. Building on this representation, we propose AtomSD, an Atom-guided Self-Distillation framework in which the learner continually extracts Procedural Atoms from its own successful trajectories and distills them into the evolving policy through Atom-conditioned supervision and task-reward optimization. Comprehensive experiments show that AtomSD consistently outperforms strong reward-based and distillation methods, exceeding the state-of-the-art baseline by up to 9.31 points in average performance. Further analyses demonstrate that the proposed Procedural Atoms provide effective guidance, highlighting the value of atom-level procedural knowledge for policy learning.
BibTeX
@misc{yu2026atomsd,
  author = {Bohan Yu and Pengfei Cao and Kai Tian and Chen Han and Shi-Yang Li and Chenxi Zhou and Junlin Liu and Cao Liu and Ke Zeng and Jun Zhao and Kang Liu},
  title = {{AtomSD: Procedural Atom-Guided Self-Distillation for LLM Agents}},
  year = {2026},
  eprint = {2610.26926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26926},
}
Figure