TL;DR
ZALT (Zero-shot Agents from Latent Topologies) enables agents to solve complex, long-horizon tasks by mapping expert demonstrations into a graph of "hub states." By planning over these abstract junctions instead of primitive actions, ZALT achieves a 55% zero-shot success rate on unseen tasks in a 3D maze, dwarfing the 6% achieved by the strongest traditional imitation learning baselines.
What Problem Does This Paper Address?
Imitation learning (IL) is a powerful way to teach agents, but it struggles with two major hurdles: compounding errors and zero-shot generalization.
In long-horizon tasks—where an agent might need to perform hundreds of actions to reach a goal—even tiny errors in a policy or dynamics model accumulate over time. Eventually, the agent drifts off-course and fails. Furthermore, collecting demonstrations for every possible start-to-goal combination is often impossible. Researchers want agents that can take an incomplete set of demonstrations and "stitch" together familiar behavioral segments to solve entirely new tasks without further training.
Methods
The authors introduce ZALT, which moves away from primitive-level planning to a hierarchical approach based on a Behavior Topology.
- Latent Space Learning: ZALT first learns a dynamics-aligned latent space using a low-level encoder-decoder model. This space is designed so that states leading to similar future behaviors are grouped together.
- Hub Discovery: The system identifies "hubs"—latent states where multiple demonstrations either converge (meet) or diverge (split). These hubs act as reusable junctions in the data.
- Topology Mapping: These hubs become nodes in a graph, and the demonstrated segments between them become edges.
- Hierarchical Planning:
- High-level: A dynamics model (using a GRU) predicts the most plausible sequence of hubs to reach a goal, given the agent's history.
- Low-level: Specific hub-to-hub policies are trained via behavioral cloning to execute the transitions between these junctions.
At inference time, ZALT searches the topology for the most likely path and executes it by chaining these learned policies.
Datasets
The authors evaluated ZALT in a custom 3D maze environment (built on MiniWorld/CARL). The task is highly compositional: the agent must find keys, unlock specific doors, collect diamonds, and deposit them in a barrel in a specific order.
The dataset consisted of:
- 18 successful demonstrations: Intentionally incomplete, covering only half of the possible start-goal pairs.
- 120 failed demonstrations: Included to help the model identify unsupported routes and common failure points.
Key Insights and Ideas
- Horizon Compression: One of ZALT's biggest wins is reducing the planning horizon. A task requiring ~250 primitive steps is compressed into roughly 33 hub-level transitions. This massive reduction (about 8x) leaves far fewer opportunities for errors to compound.
- Automatic Junction Discovery: Instead of manually defining subgoals, ZALT finds them by looking at where expert trajectories naturally intersect or branch out.
- Path-Dependent Planning: The high-level model doesn't just look at the current hub; it considers the history of hubs visited. This is crucial for tasks with internal state—like knowing if you've already picked up a specific key that isn't currently visible in the agent's first-person view.
Key Results
ZALT significantly outperformed all baselines in the zero-shot setting:
- Zero-shot Success: ZALT achieved 55% success on unseen tasks. The strongest baseline, DQN+BC, managed only 6%, while others (CQL, IQL, Option-Critic) failed completely (0%).
- Seen Task Success: Even on tasks it had seen demonstrations for, ZALT reached 72.2% success, while the best baseline only reached 11%.
- Online Adaptation: When allowed 500 episodes of online interaction to "fix" their performance, the baselines still failed to improve, highlighting how difficult long-horizon credit assignment is for standard RL/IL methods.
Limitations and Future Work
While ZALT is a major step forward, it has a few notable limitations:
- Connectivity Assumption: ZALT assumes that the demonstration data contains enough overlapping segments to form a connected graph. If a required behavioral bridge is missing, the agent cannot cross it.
- Latent Quality: The effectiveness of the hubs depends heavily on the quality of the learned latent representation. If the encoder fails to distinguish between behaviorally different but visually similar states, the topology will be flawed.
- Static Topology: Currently, ZALT builds a fixed topology from a fixed dataset. The authors suggest that future work could involve online topology mapping, where the agent adds new hubs and refines edges as it interacts with the environment.
Summary updated