Deep Implicit Imitation Reinforcement Learning in Heterogeneous Action Settings

Iason Chrysomallis, Georgios Chalkiadakis, Ioannis Papamichail, Markos PapageorgiouAAAI 2025 · DOI

TL;DR

This paper introduces HA-DIIQN, a novel deep reinforcement learning framework that enables a trainee agent to learn from a mentor agent even when their action sets are different. By developing a "bridging" mechanism to handle infeasible mentor actions, HA-DIIQN significantly accelerates learning and improves performance compared to existing methods in heterogeneous action environments.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often learn by observing expert demonstrations. Implicit imitation RL (IIRL) is a particularly useful approach because it only requires observing state transitions, not explicit actions, making it more robust to noisy or incomplete data. However, existing IIRL methods, like DIIQN, assume that the trainee agent and the mentor agent share the exact same set of possible actions (homogeneous action settings). This is a significant limitation in real-world scenarios where agents might have different physical capabilities or control mechanisms, leading to different action spaces (heterogeneous action settings). For instance, a large robot might have different motor capabilities than a smaller one, or a human operator might use a joystick while a robot uses a different control interface. This paper tackles the challenge of enabling effective imitation learning when these action spaces differ.

Methods

The core contribution is the Heterogeneous Actions Deep Implicit Imitation Q-Networks (HA-DIIQN) algorithm. It builds upon the Deep Implicit Imitation Q-Networks (DIIQN) framework but extends it to handle heterogeneous action settings.

Here's a breakdown of the key methods:

  • Action Feasibility Detection: HA-DIIQN first assesses whether a mentor's observed state transition is feasible for the trainee agent. This is done by analyzing an "error" metric (EKL) derived from the DIIQN's action estimation process. A high error suggests potential infeasibility.
  • Bridging Mechanism ("Deep k-n step repair"): When a mentor transition is deemed infeasible, HA-DIIQN employs a novel "bridging" mechanism. This algorithm searches for alternative paths within the trainee agent's experience replay buffer that can connect the trainee's current state to a state that is similar to a future state in the mentor's trajectory. This effectively creates a "bridge" to overcome the action mismatch.
  • Augmented Loss Functions: The algorithm uses modified loss functions that incorporate information from these identified "bridges" when mentor transitions are infeasible. This allows the trainee agent to learn from the mentor's guidance even when direct action replication is impossible.
  • Off-Policy Learning: The bridging mechanism is designed to retain the off-policy nature of DQN, meaning it learns from data collected under a different policy (the mentor's, indirectly) without needing to directly execute the mentor's actions.

Datasets

The paper evaluates HA-DIIQN on two environments:

  • 2D Maze Environment: A 30x30 maze environment, similar to those used in prior work on implicit imitation RL. This environment was adapted to feature distinct "regular" and "irregular" action sets for the mentor and trainee, respectively.
  • Point Maze Environment: A continuous space environment from D4RL (Fu et al. 2020), which simulates a force-actuated ball agent navigating a maze. This environment was configured with an orthogonal action set for the mentor and a distinct diagonal action set for the trainee, creating a strong heterogeneous action setting.

Key Insights and Ideas

  • Action Heterogeneity is a Major Hurdle for Imitation Learning: The paper clearly demonstrates that standard imitation learning methods struggle significantly when action spaces differ, leading to limited learning acceleration.
  • Bridging Infeasible Transitions is Crucial: The core innovation of HA-DIIQN is its ability to intelligently "bridge" over action mismatches. This allows the trainee to leverage mentor information that would otherwise be ignored or lead to errors.
  • Implicit Imitation Can Outperform the Mentor: By learning from a mentor's guidance but not being strictly bound by its actions, the trainee agent can potentially achieve better performance than the mentor itself, especially if the mentor is sub-optimal. HA-DIIQN's bridging mechanism further enhances this potential.
  • The Importance of Experience Replay: The bridging mechanism heavily relies on the trainee's experience replay buffer. This highlights the value of maintaining a rich history of the trainee's own exploration for enabling more sophisticated learning strategies.

Key Results

The paper presents experimental results on both the 2D Maze and Point Maze environments, consistently showing HA-DIIQN's superiority:

  • 2D Maze: HA-DIIQN achieved convergence significantly faster than both a baseline DQN agent and the original DIIQN agent. It converged at episode 1140, while DQN took around 3900 episodes and DIIQN around 3760 episodes.
  • Point Maze: In this more challenging continuous control task with completely distinct action sets, HA-DIIQN again demonstrated remarkable acceleration. It converged at episode 110, vastly outperforming baseline DQN (converged at 260 episodes) and DIIQN (converged at 250 episodes). The paper notes that DIIQN's performance was similar to baseline DQN because it couldn't effectively use the mentor's actions.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the "bridging" mechanism relies on the trainee agent's exploration. If the experience replay buffer is too small or the exploration is insufficient, the ability to find bridges might be limited.
  • The selection of k (observer search depth) and n (mentor search depth) for the bridging mechanism is domain-specific and requires careful tuning.
  • Future Work:
    • Enhancing Computational Efficiency: Explore modifications to the experience replay search engine, potentially using graph-based approaches, to improve efficiency, especially for large replay buffers.
  • Applying to Other Environments: Test HA-DIIQN in other MuJoCo environments and explore its application in domains like traffic signal control.
  • Integrating with Explicit Imitation: Investigate how the bridging and feasibility detection mechanisms can be integrated into explicit imitation learning algorithms.

Summary updated

BibTeX
@inproceedings{chrysomallis2025deep,
  author = {Iason Chrysomallis and Georgios Chalkiadakis and Ioannis Papamichail and Markos Papageorgiou},
  title = {{Deep Implicit Imitation Reinforcement Learning in Heterogeneous Action Settings}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33763},
}

Loading related content...

Figure