TL;DR
Standard imitation learning fails when an expert has access to "privileged" information that the learner cannot see, such as a hidden map or perfect sensors. This paper introduces ELF Distillation, a novel framework that uses a "follower" policy to evaluate expert advice and an "explorer" policy to actively seek out the missing information required to make that advice actionable.
What Problem Does This Paper Address?
In the ideal world of Imitation Learning (IL), the teacher and the student see the exact same thing. In the real world, however, experts often have "privileged information." A human driver sees further than a car’s cameras; a synthetic expert in a simulator knows the exact coordinates of every object.
The authors call these "Impossibly Good Experts." If an expert knows the location of a goal and walks straight to it, a learner who doesn't know the location will fail if they simply try to mimic the expert's path. The learner is missing the "exploratory" phase—the part where you look around to find the goal. Standard IL techniques like Behavior Cloning or DAgger often collapse in these settings because they don't account for the information gap; they try to copy the result without learning the search.
The figure above illustrates various scenarios of information asymmetry. In these examples, a hidden variable $X$ dictates the correct terminal state. The expert knows $X$ immediately at the starting state (A), allowing it to take the optimal path directly. The agent, however, must navigate to specific states to reveal $X$ before it can make the correct choice, highlighting why simply copying the expert's initial actions leads to failure.
Methods
The paper formalizes the conditions under which a learner can (or cannot) recover an optimal policy from a privileged expert. They then propose ELF Distillation (Explorer Learning from Follower).
The core idea is to train two policies simultaneously:
- The Follower ($\pi_F$): This policy's sole job is to mimic the expert using standard cross-entropy loss. Crucially, it also learns a value function ($v_F$), which estimates the long-term reward the agent would get if it followed the expert's advice from its current state.
- The Explorer ($\pi_L$): This is the policy that actually runs at test time. It is trained using Reinforcement Learning (PPO), but its reward is "shaped" by the Follower’s value function.
By using the Follower's value function as a potential-based reward signal ($r + v_F(s') - v_F(s)$), the Explorer is incentivized to move toward states where the expert’s advice is actually useful. In other words, the Explorer learns to find the "hidden" information (like the location of a key) so that it can eventually act like the expert.
Datasets
The authors evaluated their method on two challenging suites of environments designed to highlight information asymmetry:
- Minigrid: A series of grid-world tasks (e.g., Early Explore, Backtrack) where the agent must find colored balls to determine which door to open. The expert knows the ball colors in advance; the agent does not.
- Vizdoom: A 3D first-person environment (Monster Room) where the agent must look through a window to see which exit is safe from a monster. The expert knows the monster's location without looking.
Key Insights and Ideas
- The Correctness Gap: The authors prove that if an expert's advice doesn't "make sense" given the learner's limited observations, standard IL will never converge to the optimal policy.
- Expert Advice as a Map, Not a Script: Instead of treating expert actions as the "ground truth" to be copied at every step, ELF treats the expert as a guide that tells you which states are valuable.
- Value-Based Shaping: The "Aha!" moment is using the value of following the expert to drive exploration. If the Follower's value function is low, the Explorer knows it needs to look elsewhere for information before it can successfully "follow" the teacher.
Key Results
ELF Distillation consistently outperformed strong baselines, including ADVISOR, DAgger, and pure PPO.
- In the Minigrid "Early Explore" tasks, ELF was often the only method to achieve a perfect success rate, as other methods failed to deviate from the expert's "straight-to-goal" path to find the necessary colored balls.
- In Vizdoom Monster Room 2, ELF was the only method to make reliable progress. Baselines like ADVISOR tended to "latch" onto the expert's predictable signal without understanding the underlying reason (the monster's location), leading to death when they finally had to make a choice.
Limitations and Future Work
The primary limitation is the bias in the follower's value estimate. Early in training, the Follower hasn't seen enough states to provide an accurate value function. If the Follower provides a poor estimate, it can lead the Explorer into a "dead zone" where it stops exploring useful areas.
The authors suggest that future work could involve more sophisticated training schedules—perhaps training the Follower more heavily in the early stages—or developing methods to automatically detect when the Follower's value function is reliable enough to guide the Explorer.
Summary updated