Imitation Learning via Focused Satisficing

Rushit N. Shah, Nikolaos Agadakos, Synthia Sasulski, Ali Farajzadeh, Sanjiban Choudhury, Brian D. Ziebart

Abstract

Traditional imitation learning often assumes that human demonstrators are optimal, attempting to recover a single reward function that explains their behavior. However, humans frequently "satisfice"—aiming for performance that is "good enough" rather than mathematically optimal. This paper introduces Minimally Subdominant Focused Imitation (MinSubFI), a novel approach that shifts the objective from optimality to guaranteed acceptability. By minimizing "subdominance" across a vector of cost features, MinSubFI ensures that an agent's behavior remains within the demonstrator's acceptable set, providing robustness to noisy or suboptimal demonstrations.

Introduction: The Challenge of Satisficing

Standard imitation learning (IL) typically relies on Behavioral Cloning (BC) or Inverse Reinforcement Learning (IRL). Both frameworks assume the demonstrator is an expert optimizer. In practice, human demonstrations are often suboptimal, characterized by noise, fatigue, or a simple desire to meet a threshold of performance. For instance, a pilot might prioritize a safe landing over a perfectly smooth one. Standard IL agents may incorrectly learn that the "jerkiness" of a novice's landing is a desired feature. The core challenge addressed here is learning a policy that can exceed the demonstrator's performance by identifying and adhering to their underlying criteria for acceptability without requiring an explicit, scalar reward function.

Methodology: Minimally Subdominant Focused Imitation

The authors propose MinSubFI, which utilizes a vector of cost features (e.g., fuel efficiency, safety, smoothness) instead of a single scalar reward. This multi-objective approach allows the agent to define an "acceptable set" based on the demonstrator's performance.

Comparison of traditional Inverse Reward Learning (top) and the proposed Subdominance Minimization (bottom) architectures.

The technical framework relies on several key innovations:

  • Subdominance: A margin-based metric measuring how far an imitator's trajectory is from Pareto-dominating the demonstration. Zero subdominance implies the agent is at least as good as the demonstrator in every cost feature.
  • MinSubFI Algorithm: A policy gradient method that uses subdominance as its reinforcement signal, seeking a policy that is maximally acceptable.
  • Snippet-focused Learning: To handle long, inconsistent demonstrations, the method extracts and optimizes over high-quality trajectory snippets, allowing the agent to ignore suboptimal segments.
  • Latent Cost Features (MinSubFI-LCF): A method to learn latent feature representations from pairwise preferences, reducing the need for manual feature engineering.

Theoretical Framework: Acceptability over Optimality

The shift from optimality to satisficing allows the agent to be "better" than the demonstrator by simply staying within the bounds of what the demonstrator deemed acceptable.

The theoretical framework of acceptable sets. The left plot shows the Pareto frontier and the demonstrator's acceptable set. The right plot illustrates how rollouts are evaluated against the demonstration to minimize subdominance.

By using a vector-based approach, MinSubFI avoids the "reward bottleneck" where excelling in one metric (like speed) might come at the expense of another (like safety). If a demonstration is poor, it is easily subdominated, which naturally guides the agent toward superior policies.

Experimental Evaluation

The researchers evaluated MinSubFI across diverse benchmarks, including classic control (Cartpole, LunarLander), MuJoCo robotics (Hopper, HalfCheetah, Walker2d), and human-collected trajectories. To test robustness, they also used synthetic suboptimal demonstrations generated by partially trained PPO agents.

Key Results

  • Guaranteed Acceptability: MinSubFI variants achieved significantly higher "relative satisficing" values compared to baselines like GAIL or T-REX. It was often the only method to consistently guarantee acceptable behavior.
  • Robustness to Noise: On human LunarLander data, MinSubFI outperformed BC and AIRL, which struggled with irregular human input.
  • Stability with Poor Data: When trained on the "worst 60%" of demonstrations, MinSubFI's performance remained stable, while methods like T-REX degraded significantly.

Limitations and Future Directions

While promising, the authors note that the core method still benefits from well-defined cost features, though the LCF variant mitigates this. Future work will focus on more general snippet generation strategies and exploring how to adapt to subjective, time-varying "aspiration levels" in human teachers.

Summary updated

BibTeX
@inproceedings{shah2025imitation,
  author = {Rushit N. Shah and Nikolaos Agadakos and Synthia Sasulski and Ali Farajzadeh and Sanjiban Choudhury and Brian D. Ziebart},
  title = {{Imitation Learning via Focused Satisficing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/975},
}
Figure