TL;DR
Traditional active imitation learning reduces expert effort by asking for demonstrations where the robot's policy is most uncertain. This paper introduces Adaptive Agents via Latent Topologies (AALT), which instead focuses on composability by building a latent map of known behaviors and requesting short "bridge" demonstrations that maximize overall start-to-goal connectivity. In a simulated robotic retrieval task, AALT achieves 100% success across all tasks using just 3 short expert queries, drastically outperforming traditional policy-centric methods.
What Problem Does This Paper Address?
In multi-task robotic domains, the number of possible start-to-goal configurations can grow combinatorially. However, many of these tasks share underlying, reusable behaviors—such as moving an obstacle out of the way before retrieving an object.
Standard active imitation learning (IL) methods are typically policy-centric: they ask the expert for full-task demonstrations based on where the learner's policy has the highest uncertainty about what action to take next. The problem? These methods don't explicitly value how well a new demonstration can be composed with existing knowledge. As a result, an expert might be asked to provide multiple long, redundant demonstrations that substantially overlap, wasting expensive human effort.
Methods
AALT shifts the active learning objective from policy uncertainty to reachability information gain. The method operates in three main phases:
- Topology Construction (Pre-training): AALT embeds demonstrated states into a compact latent space shaped by action effects using forward and inverse dynamics models. It identifies hub states—points in the latent space where trajectories naturally converge or diverge (e.g., a specific shelf configuration after a common obstacle is moved). The demonstrated behaviors between these hubs form the directed edges of a latent topology.
- Composition-Aware Acquisition: Instead of asking for a full task, AALT evaluates missing edges (potential "bridges") between existing hubs. It greedily requests the single bridge demonstration that maximizes the expected increase in start-to-goal connectivity across the entire target task distribution.
- Inference & Execution: At test time, AALT uses Dijkstra’s algorithm to find the most reliable path through the latent topology from the start state to the goal. A goal-conditioned diffusion policy then executes the path one edge at a time, navigating from hub to hub.
Datasets
The method was evaluated on a newly designed Toolshelf domain:
- The Setup: A simulated UR5e robot arm tasked with retrieving three ordered part canisters from a shelf containing movable access guards (blockers).
- Task Space: 72 total tasks created by crossing 6 distinct starting shelf layouts with 12 unique multi-canister work orders.
- Initial Data: 24 successful demonstrations (196 transitions) where the shelf was already perfectly staged for specific work orders. Crucially, 48 cross-family combinations (e.g., a specific layout paired with an unseen work order family) were completely omitted to test the agent's ability to generalize via active learning.
Key Insights and Ideas
- The Power of Bridges: A short, simple action (like moving a single blocking canister to connect two parts of a shelf layout) can instantly unlock dozens of tasks simultaneously if it bridges two previously disconnected sub-graphs of behavioral knowledge.
- Theoretical Grounding: The authors formally prove that reachability information gain factorizes into AALT’s connectivity gain and the uncertainty of bridge acquisition. This provides a rigorous mathematical justification for prioritizing connectivity over policy uncertainty.
- Action-Shaped Latent Spaces: Visual similarity alone is deceptive for planning. By training the latent space using inverse dynamics, AALT ensures that states are clustered based on what the robot can do from them, rather than just what they look like.
Key Results
AALT demonstrated massive efficiency gains over state-of-the-art active learning baselines:
- 100% Success in 3 Queries: AALT consistently achieved perfect success (72/72 tasks) using just 3 short bridge demonstrations totaling only 5 state transitions beyond the initial dataset.
- Baseline Comparison: The strongest policy-centric baseline, Active Multi-task Fine-tuning (AMF-Full), only reached 46.9% success after 20 full-task demonstrations (171.6 transitions), largely because it wasted queries repeating variations of individual tasks it already partially understood.
Limitations and Future Work
- Reliance on Initial Structure: AALT cannot currently propose entirely new internal hub states; it can only connect hubs that are discovered within the initial dataset or explicitly provided via start/goal states.
- Greedy Selection: The current acquisition rule is one-step greedy. It can miss complementary bridge sets—pairs of bridges that unlock massive connectivity together but offer little value individually. The authors suggest multi-step lookahead or beam search as future solutions.
- Skill Refinement: AALT assumes that if a route exists in the topology, it can be executed. If an existing edge behavior is poorly learned and unreliable, AALT won't request a demonstration to fix it. A promising future direction is a hybrid objective combining AALT's connectivity gain with policy-centric uncertainty refinement.
Summary updated