TL;DR
EvoSkill-S1 is a novel reinforcement learning (RL) framework that treats reusable problem-solving skills as objects of evolution. By applying genetic-inspired operators—like mutation and crossover—to a "skill bank," the system iteratively transforms weak strategies into robust, abstract procedural guidance, leading to significant performance boosts in complex mathematical reasoning and code generation.
What Problem Does This Paper Address?
While Large Language Model (LLM) agents have become adept at reasoning, they often struggle with "hard-negative" failures—problems they consistently miss despite having access to previous experiences. Most current methods for maintaining "skill banks" (collections of strategies and tips) are passive: they simply append new experiences or use basic heuristics to summarize them. This often results in shallow, redundant, or overly specific updates that don't generalize well.
The authors of EvoSkill-S1 argue that skills shouldn't just be accumulated; they should be evolved. The challenge lies in how to systematically repair brittle strategies, combine complementary partial skills, and ensure that evolved skills remain abstract enough to be useful for unseen problems.
Methods
The EvoSkill-S1 pipeline moves beyond simple reflection by implementing a dedicated, RL-trained Skill Evolver.
- Iterative Skill Evolution: The framework identifies "hard negatives" (problems the agent failed). It retrieves "parent" skills from the bank and uses the Skill Evolver to generate "child" skills.
- Genetic-Inspired Operators: Instead of unconstrained reflection, the evolver uses five specific operators:
- Revision: Small, conservative repairs to a strategy.
- Cross-revision: Borrowing a specific "motif" or safeguard from a related problem.
- Mutation: Introducing a completely alternative algorithmic approach.
- Crossover: Combining the "backbone" of one parent with a local improvement from another.
- Dual-reflection: Strengthening correctness-critical safeguards.
- RL Training (GRPO): The evolver is trained using Group Relative Policy Optimization (GRPO). It learns to favor child skills that actually improve the performance of a fixed base solver on downstream tasks.
- Skill Bank Fusion: Since different operators solve different types of failures, the framework fuses operator-specific banks into a single, high-quality collection.
- Procedural Skill Interface (PSI): To prevent evolved skills from being too tied to their source problems, the PSI "proceduralizes" them—stripping away specific constants and stories to create abstract, retrieval-ready guidance.
Datasets
The authors evaluated EvoSkill-S1 across a rigorous suite of math and code benchmarks:
- Math: AIME24, AIME25, AMC23, Minerva Math, and OlympiadBench.
- Code: APPS (test set) and LiveCodeBench.
- Training/Evolution Pools: Hard subsets mined from DAPO (math) and APPS-train (code), totaling roughly 4,000 math and 2,200 code problems.
Key Insights and Ideas
- Skills as Genetic Material: Treating skills as entities that can be mutated and recombined is more effective than treating them as static diary entries of past failures.
- The Power of RL in Evolution: Training the evolver with RL (GRPO) significantly increases the "yield" of useful skills. As shown in Figure 4, RL-trained evolvers solve more hard negatives and produce more high-quality "write-back" candidates than those using vanilla LLM prompting.
- Operator Complementarity: No single genetic operator is a silver bullet. Some failures require the "conservative repair" of Revision, while others require the "radical shift" of Mutation. This justifies the "Fusion" approach.
- Abstraction is Mandatory: Evolution often produces skills cluttered with details from the specific problem that triggered the update. The "Proceduralization" step is critical for ensuring these skills actually help when the agent encounters a brand-new problem.
Key Results
EvoSkill-S1 consistently outperformed both initialized skill banks and strong external baselines like Reflexion and ExpeL-SI.
- Performance Gains: The framework achieved relative improvements of up to 12.39% over the initial skill banks.
- Superiority over Baselines: Compared to the strongest external baseline without direct skill evolution, the relative improvement reached as high as 20.39%.
- Efficiency: The RL-based training allowed the system to recover more hard negatives under a fixed "write-back" budget (the number of new skills allowed into the bank).
Limitations and Future Work
- Fixed Solver: Currently, the base solver (the model doing the actual math/coding) remains frozen. The authors suggest that co-evolving the solver and the skill bank simultaneously could unlock even higher performance.
- Computational Overhead: While the evolution happens "offline," the process of generating and evaluating multiple candidate child skills for every failure is computationally intensive.
- Domain Expansion: The current work focuses on math and code, where "correctness" is easily verifiable. Applying this to more subjective domains (like creative writing or general planning) would require more complex reward modeling.
Summary updated