TL;DR
AtomSD introduces a modular way for LLM agents to learn "how to act" by breaking down successful experiences into fine-grained Procedural Atoms (Intent, When, Do, and Avoid). By continually extracting these atoms from its own successes and distilling them back into its policy, the agent learns to compose complex behaviors that generalize far better than traditional trajectory-based or skill-bundling methods.
What Problem Does This Paper Address?
Large Language Models (LLMs) are great at "knowing that" (declarative knowledge) but often struggle with "knowing how" (procedural knowledge). An LLM might know the chemical properties of reagents but fail to execute a multi-step lab experiment in the correct order.
Current methods to fix this usually involve:
- Full Trajectories: Showing the model a complete successful run. These are often too specific to one scenario and hard to reuse.
- Bundled Skills: Grouping actions into "skills." These can be too rigid; a new task might only need half of a pre-defined skill.
The authors argue that we need a more modular, "atomic" unit of procedural knowledge that can be flexibly recombined across different tasks and contexts.
Methods
The researchers propose AtomSD (Atom-guided Self-Distillation), which centers on two main innovations:
1. The Procedural Atom
Instead of a long list of actions, knowledge is represented as an Atom consisting of four parts:
- Intent: What is the goal?
- When: Under what conditions is this applicable?
- Do: What is the specific sequence of operations?
- Avoid: What are the common failure modes or "trap" actions to steer clear of?
2. The Self-Distillation Loop
AtomSD uses a "student-teacher" framework to internalize these atoms:
- Online Abstraction: The student agent attempts tasks. When it succeeds, it acts as its own "knowledge extractor," turning the successful trajectory into a set of Procedural Atoms.
- Atom-Guided Supervision: A teacher model (an Exponential Moving Average of the student) is given these extracted atoms as context. It then provides token-level guidance to the student on how to solve the task.
- Joint Optimization: The student is updated using a combination of environment rewards (via GRPO) and a distillation loss that encourages it to match the teacher's atom-guided logic. Over time, the student "internalizes" the atoms, meaning it can perform the procedures even when the explicit atom guidance is removed.
Datasets
The framework was evaluated across four diverse benchmarks requiring complex planning:
- ALFWorld: Household planning and execution in text environments.
- ScienceWorld: Multi-step scientific experiments (e.g., thermodynamics, biology).
- WebShop: Navigating a web interface to find products under constraints.
- StableToolBench: Multi-turn tool use and API calling.
Key Insights and Ideas
- The Power of "Avoid": One of the most significant findings is that explicitly representing what not to do is crucial. Removing the "Avoid" component from the atoms led to a massive performance drop (up to 14 points).
- Compositionality is King: Because atoms are fine-grained, the model can learn an atom in one task (e.g., "how to heat an object") and apply it in a completely different task (e.g., "making dough").
- Efficiency: Atoms are much more compact than full trajectories. On average, an atom uses only 0.15k tokens, whereas a full trajectory might take 4.97k. This makes them highly efficient for providing context during training.
- Co-Evolution: As the student gets better, it extracts higher-quality atoms, which in turn makes the distillation process more effective. This creates a virtuous cycle of improvement.
Key Results
AtomSD consistently outperformed strong baselines, including standard Reinforcement Learning (GRPO) and other distillation methods (OPSD, Skill-SD).
- Performance Gains: AtomSD exceeded the state-of-the-art baseline by up to 9.31 points in average performance across all benchmarks.
- Scaling Up: Using a Qwen3.5-9B backbone, AtomSD-9B actually outperformed DeepSeek-V4-Pro and came within 5.4 points of GPT-5.5 (a much larger, API-based model).
- Generalization: The model showed strong performance on "unseen" tasks in ScienceWorld and ALFWorld, proving that it isn't just memorizing trajectories but learning reusable procedural logic.
Limitations and Future Work
- The "Cold Start" Problem: AtomSD relies on the student being able to find some successful trajectories to start the extraction process. If a task is so hard that the model never succeeds initially, there are no atoms to extract.
- Extraction Quality: The process depends on the LLM's ability to accurately "summarize" its own actions into the Atom format. If the extraction prompt is poorly handled, the resulting guidance could be noisy.
- Future Directions: The authors suggest exploring multi-agent settings where different agents could share a global "Atom Library," or applying this to even more complex, long-horizon reasoning tasks.
Summary updated