SkillEarth: Teaching General-Purpose Agents to Act on Earth via Skill Evolution

Haojun Tang, Wenda Zhao, Xiang Zhou, Qiwei Lin, Lida Huang, Yan WangarXiv 2026 · arXiv

TL;DR

SkillEarth is a self-evolving framework that enables general-purpose AI agents to master complex remote sensing (RS) tasks by automatically extracting and refining reusable "skills" from execution trajectories. By shifting from holistic, outcome-based reflection to component-level, observation-grounded evolution, it allows agents to adapt to new tools and workflows without expert intervention or model retraining.

What Problem Does This Paper Address?

Remote sensing—using satellite or aerial imagery to monitor Earth—is inherently complex. Tasks like disaster assessment or urban planning aren't solved with a single prompt; they require multi-step workflows involving specialized tools for GIS processing, spectral analysis, and object detection.

Current RS agents face a "brittleness" problem. They typically rely on either hard-coded expert knowledge or expensive model fine-tuning. If a new tool is introduced or a workflow changes, these agents often fail. While "skill extraction" (learning from past successes) is a known strategy, existing methods treat skills as monolithic blocks of text. When a complex multi-step plan fails, these methods struggle to pinpoint which part of the plan was wrong, leading to coarse, unreliable refinements.

Methods

The authors propose SkillEarth, which treats skill acquisition as an evolutionary process. The framework consists of four main stages:

  1. Component-wise Skill Extraction: Instead of a single "blob" of text, SkillEarth decomposes a skill into independently editable components: name, description, activation conditions, execution plan, and Grounded Observation Routing (GOR).
  2. Observation-Grounded Evolution: This is the core innovation. An "Evolver" executes the skill in a real tool environment. It uses a dual-judge system:
  3. Tool Judge: Validates the syntax, order, and logic of tool calls.
  4. Semantic Judge: Evaluates if the final output actually answers the user's question.
  5. GOR Graph Construction: For every successful run, the system builds a Grounded Observation Routing graph. This graph explicitly maps how the output of one tool (e.g., a boundary file) becomes the input for the next (e.g., a spectral index calculator). This "data flow" map is then translated into natural language rules.
  6. Skill Decision & Bank: A "Decider" manages a library of skills (the Skill Bank), determining whether to add a new skill, merge it with an existing similar one, or discard it if it's redundant.
Comparison of SkillEarth's component-level refinement versus traditional holistic refinement.

Datasets

The researchers evaluated SkillEarth on two primary benchmarks:

  • OpenEarth-Bench: A large-scale dataset containing over 14,000 training trajectories and 24 specialized RS tools across categories like perceptual, GIS, and spectral analysis.
  • Earth-Bench: A benchmark focused on cross-modal tasks under both automatic planning and instruction-following settings, featuring 248 expert-curated samples.

Key Insights and Ideas

  • Data Flow is King: In complex toolchains, knowing what tool to call is only half the battle. Knowing how to pass data between them (the "routing") is often where agents fail. SkillEarth’s GOR graph makes this implicit knowledge explicit.
  • Component-Level > Holistic: By breaking skills into parts, the system can fix a specific routing error without accidentally breaking a perfectly good execution plan.
  • Cross-Model Transferability: One of the most striking findings is that skills are model-agnostic. A skill evolved using a "frontier" model like GPT-4 can be retrieved and used by a much smaller, open-source model (like Qwen-4B) to significantly boost its performance.
  • Generalization via Placeholders: During evolution, the system replaces specific coordinates or dates with typed placeholders (e.g., [location], [date]), turning a one-off success into a generalized template.
Figure 3. Overview of SkillEarth. (a) The Extractor constructs component-wise skills by overwriting templates with LLM-inferred components. (b) The Evolver validates and refines skills using dual-judge feedback and GOR graphs. (c) The Decider merges, discards, or adds skills to the skill bank. (d) During application, the Executor uses retrieved skills to generate final answers.

Key Results

SkillEarth consistently outperformed state-of-the-art RS agents and other skill extraction methods:

  • Superior Accuracy: On OpenEarth-Bench, it achieved the best performance on 8 out of 9 end-to-end metrics, significantly improving tool selection and execution consistency.
  • Efficiency Gains: On Earth-Bench, it improved "Exact Match" (matching the expert trajectory) by 3.75% in automatic planning and a massive 13.00% in instruction-following modes compared to the next best method.
  • Qualitative Improvement: As shown in Figure 4, evolved skills move from "over-specified" plans (hard-coded steps) to "generalizable" plans that describe the analytical intent, making them much more robust to new queries.
Figure 4. Qualitative analysis of a skill before and after evolution. (a) The initial extracted skill contains over-specified task planning and ambiguous observation routing. (b) The evolved skill adopts generalizable task planning and precise output-to-argument routing.

Limitations and Future Work

  • Reasoning Bottleneck: While SkillEarth improves tool use, the final synthesis of an answer still relies on the base LLM’s inherent reasoning capabilities. If the model can't interpret the tool's output correctly, the skill can't save it.
  • Initial Trajectories: The system currently requires some initial demonstrated trajectories to start the extraction process. Future work could explore "cold-start" scenarios where the agent explores the tool space entirely on its own.
  • Real-time Evolution: The current evolution process happens "offline" to build the Skill Bank. Integrating this into a real-time, continuous learning loop during deployment is a promising next step.

Summary updated

Abstract

Recently developed remote sensing (RS) agents execute real-world tasks with complex workflows through multi-step tool invocation. These agents, however, typically rely on predefined expertise or parameter updates. They thus require updating expert knowledge or retraining models whenever new tools, tasks, or execution patterns emerge, limiting their continual adaptability in dynamic Earth environments. In this work, we seek to equip general-purpose agents with RS tool-call knowledge that can be automatically extracted from demonstrated trajectories without expert intervention or parameter updates. Specifically, we propose SkillEarth, a self-evolving skill extraction method that externalizes RS tool-call knowledge as reusable, retrievable, and evolvable skill files and refines them through observation-grounded evolution. Unlike existing automatic skill extraction methods that rely on outcome-based reflection to refine skills holistically, SkillEarth grounds skill evolution in real tool-chain execution and combines dual-judge feedback with a grounded observation routing graph for precise component-level refinement. The resulting skills are generalizable, reliable, and transferable: they improve overall RS tool-call performance across three frontier LLMs and four open-source LLMs, are refined through real tool execution feedback, and support cross-model reuse without retraining. Experiments on OpenEarth-Bench and Earth-Bench demonstrate that SkillEarth achieves better tool selection, argument correctness, execution-order consistency, and task completion than existing RS agents and skill extraction methods.
BibTeX
@misc{tang2026skillearth,
  author = {Haojun Tang and Wenda Zhao and Xiang Zhou and Qiwei Lin and Lida Huang and Yan Wang},
  title = {{SkillEarth: Teaching General-Purpose Agents to Act on Earth via Skill Evolution}},
  year = {2026},
  eprint = {2611.15481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15481},
}
Figure