Imitation Learning via Focused Satisficing

Rushit N. Shah, Nikolaos Agadakos, Synthia Sasulski, Ali Farajzadeh, Sanjiban Choudhury, Brian D. Ziebart

TL;DR

This paper introduces "Focused Satisficing" for imitation learning, a new approach that trains AI agents to be "good enough" according to human-like aspiration levels, rather than strictly optimal. By minimizing "subdominance," their method, MinSubFI, achieves higher demonstrator acceptability and more robust performance, especially with imperfect or noisy human demonstrations.

What Problem Does This Paper Address?

Traditional imitation learning (IL) methods often assume that human demonstrations are near-optimal according to some underlying, fixed cost function. However, this assumption breaks down when dealing with real-world human behavior, which is often "satisficing" – meaning it's acceptable rather than perfectly optimal. Humans might choose a good-enough solution that meets their current aspiration level, which can dynamically change (e.g., due to fatigue or improving skill). When an AI agent tries to perfectly mimic these satisficing behaviors, it can lead to "value misalignment," where the agent's actions, while technically matching the demonstration, might not be truly acceptable to the human demonstrator in a broader sense. This paper addresses the challenge of training AI agents that can learn from potentially suboptimal or noisy human demonstrations and produce behaviors that are genuinely acceptable to the demonstrator, even when the demonstrator's exact acceptability criteria are unknown and potentially variable.

Methods

The core innovation is Focused Satisficing, a new paradigm for imitation learning. Instead of trying to perfectly replicate demonstrations or learn an exact reward function, the goal is to make the imitator's behavior "acceptable" to the demonstrator. This is achieved by minimizing subdominance, a measure of how far an imitator's trajectory is from Pareto-dominating a demonstrator's trajectory across multiple cost features.

Key techniques include:

  • Minimally Subdominant Focused Imitation (MinSubFI): This is the proposed algorithm. It uses a margin-based objective to guide deep reinforcement learning.
  • Subdominance: A metric that quantifies the insufficiency of an imitator's trajectory compared to a demonstrator's, measured across different cost features. The goal is to make this subdominance as low as possible.
  • Snippet-focused Subdominance: An extension that allows the model to focus on high-quality "snippets" (portions) of trajectories, even if the overall trajectory is less impressive. This helps capture acceptability at a more granular level.
  • Policy Gradient Optimization: MinSubFI leverages policy gradient methods, treating subdominance as a reinforcement signal to update the policy.
  • Learned Cost Features: The paper also explores learning a representation of cost features from pairwise preferences over demonstrations, reducing the need for hand-engineered features.

Datasets

The paper uses a mix of standard robotics and control environments from OpenAI Gym:

  • Classic Control: Cartpole, Lunar Lander
  • MuJoCo: Hopper, HalfCheetah, Walker

For training and evaluation, they use:

  • Synthetic Demonstrations: Generated by a suboptimal PPO policy.
  • Human Demonstrations: Collected from non-expert human players for the Lunar Lander environment.

The paper does not introduce new benchmark datasets but uses existing environments to evaluate their method against established baselines.

Key Insights and Ideas

  • Satisficing is Key: The paper argues that human behavior is often "satisficing" (good enough) rather than "optimizing" (perfectly optimal). This fundamental shift in perspective is crucial for understanding and replicating human-like behavior.
  • Acceptability Over Optimality: The core idea is that AI agents should aim for demonstrator acceptability rather than strict optimality. This is a more realistic and achievable goal when learning from human data.
  • Subdominance as a Proxy for Acceptability: Minimizing subdominance provides a principled way to ensure that the imitator's behavior is acceptable across a range of potential demonstrator preferences, even if those preferences are unknown.
  • Robustness to Imperfect Data: MinSubFI shows remarkable robustness to noisy or suboptimal demonstrations, outperforming other methods when the quality of training data degrades.
  • Snippet-Level Learning: Focusing on high-quality snippets within demonstrations allows the agent to learn valuable behaviors even from otherwise imperfect trajectories.

Key Results

  • Higher Acceptability: MinSubFI consistently achieves significantly higher rates of demonstrator acceptability (y-satisficing values) compared to baselines like BC, TREX, and GAIL across various environments. For example, in cartpole, MinSubFI achieves a y-satisficing value of 2.68, while BC achieves only 0.19.
  • Competitive True Returns: Despite focusing on acceptability, MinSubFI achieves competitive or superior true episode returns compared to demonstrations and baselines, especially in challenging environments.
  • Robustness to Demonstration Quality: MinSubFI's performance degrades much less than T-REX and AIRL when trained on progressively lower-quality subsets of demonstrations.
  • Effective with Human Data: MinSubFI significantly outperforms baselines when trained on noisy human demonstrations for the Lunar Lander task.
  • Learned Features Improve Performance: Using learned cost features (MinSubFILCF) further boosts performance, suggesting that explicit knowledge of the true cost function is not always necessary.

Limitations and Future Work

  • Computational Cost: While offline bootstrapping improves sample efficiency, training can still be computationally intensive.
  • Generalization Guarantees: While MinSubFI provides theoretical guarantees for acceptability, these rely on certain assumptions about cost features and the convexity of regions.
  • Future Directions:
    • Learning latent representations of cost features without explicit human input.
  • Developing more general snippet generation strategies.
  • Exploring methods for guaranteeing higher levels of demonstrator acceptability.
  • Applying the method to domains lacking explicit true return functions or where acceptability is dynamic and subjective.

Summary updated

BibTeX
@inproceedings{shah2025imitation,
  author = {Rushit N. Shah and Nikolaos Agadakos and Synthia Sasulski and Ali Farajzadeh and Sanjiban Choudhury and Brian D. Ziebart},
  title = {{Imitation Learning via Focused Satisficing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/975},
}
Figure