TL;DR
SkillEarth is a self-evolving framework that enables general-purpose AI agents to master complex remote sensing (RS) tasks by automatically extracting and refining reusable "skills" from execution trajectories. By shifting from holistic, outcome-based reflection to component-level, observation-grounded evolution, it allows agents to adapt to new tools and workflows without expert intervention or model retraining.
What Problem Does This Paper Address?
Remote sensing—using satellite or aerial imagery to monitor Earth—is inherently complex. Tasks like disaster assessment or urban planning aren't solved with a single prompt; they require multi-step workflows involving specialized tools for GIS processing, spectral analysis, and object detection.
Current RS agents face a "brittleness" problem. They typically rely on either hard-coded expert knowledge or expensive model fine-tuning. If a new tool is introduced or a workflow changes, these agents often fail. While "skill extraction" (learning from past successes) is a known strategy, existing methods treat skills as monolithic blocks of text. When a complex multi-step plan fails, these methods struggle to pinpoint which part of the plan was wrong, leading to coarse, unreliable refinements.
Methods
The authors propose SkillEarth, which treats skill acquisition as an evolutionary process. The framework consists of four main stages:
- Component-wise Skill Extraction: Instead of a single "blob" of text, SkillEarth decomposes a skill into independently editable components: name, description, activation conditions, execution plan, and Grounded Observation Routing (GOR).
- Observation-Grounded Evolution: This is the core innovation. An "Evolver" executes the skill in a real tool environment. It uses a dual-judge system:
- Tool Judge: Validates the syntax, order, and logic of tool calls.
- Semantic Judge: Evaluates if the final output actually answers the user's question.
- GOR Graph Construction: For every successful run, the system builds a Grounded Observation Routing graph. This graph explicitly maps how the output of one tool (e.g., a boundary file) becomes the input for the next (e.g., a spectral index calculator). This "data flow" map is then translated into natural language rules.
- Skill Decision & Bank: A "Decider" manages a library of skills (the Skill Bank), determining whether to add a new skill, merge it with an existing similar one, or discard it if it's redundant.
Datasets
The researchers evaluated SkillEarth on two primary benchmarks:
- OpenEarth-Bench: A large-scale dataset containing over 14,000 training trajectories and 24 specialized RS tools across categories like perceptual, GIS, and spectral analysis.
- Earth-Bench: A benchmark focused on cross-modal tasks under both automatic planning and instruction-following settings, featuring 248 expert-curated samples.
Key Insights and Ideas
- Data Flow is King: In complex toolchains, knowing what tool to call is only half the battle. Knowing how to pass data between them (the "routing") is often where agents fail. SkillEarth’s GOR graph makes this implicit knowledge explicit.
- Component-Level > Holistic: By breaking skills into parts, the system can fix a specific routing error without accidentally breaking a perfectly good execution plan.
- Cross-Model Transferability: One of the most striking findings is that skills are model-agnostic. A skill evolved using a "frontier" model like GPT-4 can be retrieved and used by a much smaller, open-source model (like Qwen-4B) to significantly boost its performance.
- Generalization via Placeholders: During evolution, the system replaces specific coordinates or dates with typed placeholders (e.g.,
[location],[date]), turning a one-off success into a generalized template.
Key Results
SkillEarth consistently outperformed state-of-the-art RS agents and other skill extraction methods:
- Superior Accuracy: On OpenEarth-Bench, it achieved the best performance on 8 out of 9 end-to-end metrics, significantly improving tool selection and execution consistency.
- Efficiency Gains: On Earth-Bench, it improved "Exact Match" (matching the expert trajectory) by 3.75% in automatic planning and a massive 13.00% in instruction-following modes compared to the next best method.
- Qualitative Improvement: As shown in Figure 4, evolved skills move from "over-specified" plans (hard-coded steps) to "generalizable" plans that describe the analytical intent, making them much more robust to new queries.
Limitations and Future Work
- Reasoning Bottleneck: While SkillEarth improves tool use, the final synthesis of an answer still relies on the base LLM’s inherent reasoning capabilities. If the model can't interpret the tool's output correctly, the skill can't save it.
- Initial Trajectories: The system currently requires some initial demonstrated trajectories to start the extraction process. Future work could explore "cold-start" scenarios where the agent explores the tool space entirely on its own.
- Real-time Evolution: The current evolution process happens "offline" to build the Skill Bank. Integrating this into a real-time, continuous learning loop during deployment is a promising next step.
Summary updated