Case-Based Inverse Reinforcement Learning Using Temporal Coherence

Jonas Nüßlein, Steffen Illium, Robert Müller, Thomas Gabor, Claudia Linnhoff-Popien

Abstract

Traditional Imitation Learning (IL) often suffers from high data requirements and a dependency on expert action labels, which are frequently unavailable in real-world scenarios. This paper introduces Case-Based Inverse Reinforcement Learning (CB-IRL), an approach designed for Learning from Observation (LfO). By leveraging temporal coherence and a case-based reasoning framework, CB-IRL enables agents to recover complex behaviors from sparse expert demonstrations—requiring as little as 10% of the data used by state-of-the-art baselines—by focusing on high-level state-reaching strategies rather than low-level action mimicry.

1. Introduction

Imitation Learning (IL) allows agents to acquire skills by observing expert demonstrations. However, two primary bottlenecks hinder its widespread application: action dependency and data volume. Most IL methods require access to the specific actions taken by the expert, yet in many cases (such as learning from video), only state transitions are visible. Furthermore, adversarial methods like Generative Adversarial Imitation Learning (GAIL) typically require a vast number of trajectories to achieve convergence.

CB-IRL addresses these challenges by reframing imitation as a state-reaching task. Instead of attempting to replicate every micro-movement of an expert, the agent learns to navigate toward "target state areas" identified from expert data. This shift in perspective allows the agent to succeed even when expert data is "gappy" or incomplete.

2. Methodology

2.1 The Equality Net ($E$)

At the core of CB-IRL is the Equality Net, a neural network trained to act as a similarity judge. Rather than using a standard reward function or regressing the exact temporal distance between states, the Equality Net performs binary classification. It predicts whether two states are "temporally coherent"—meaning they occur within a specific windowFrame of each other in a trajectory. This classification approach provides a more stable training signal than regression, avoiding the "moving target" problem often encountered in Reinforcement Learning.

2.2 The Case Base ($C$) and Reward Mechanism

The Case Base serves as a library of expert states. During the training phase, the agent’s current state is compared against the states stored in the Case Base using the Equality Net. If the agent reaches a state deemed similar to an expert state, it is rewarded.

The reward is proportional to the expert state's progress along the original trajectory; states further along the path yield higher rewards. This structure creates an implicit curriculum: the agent first learns to reach the early stages of a task and gradually "unlocks" higher rewards as it masters the ability to reach states further down the expert's path.

3. Experimental Setup

The effectiveness of CB-IRL was evaluated across four standard OpenAI Gym environments, modified to test state-reaching capabilities:

  • Acrobot: A 2-link pendulum system.
  • Mountain Car: A classic under-actuated physics problem.
  • Lunar Lander: A navigation and landing task.
  • Half Cheetah Discrete: A modified version of the Half Cheetah environment where absolute x-position data was included to define a state-reaching objective, and the action space was discretized to simplify exploration.

4. Key Insights and Results

4.1 Data Efficiency and Sparsity

CB-IRL demonstrates remarkable efficiency, achieving near-optimal performance using only one-tenth of the expert states required by GAIfO (Generative Adversarial Imitation from Observation). The model proved robust to "gappy" data; even when 90% of expert steps were removed, the agent successfully recovered the high-level strategy required to complete the task.

4.2 Stability and Strategy

By focusing on temporal coherence rather than exact action matching, CB-IRL avoids the instability often seen in adversarial training. In the Half Cheetah domain, CB-IRL exhibited smoother learning curves and significantly less "forgetting" than the GAIfO baseline. The results suggest that by forcing the agent to work with incomplete trajectories, it is incentivized to learn the underlying strategy (e.g., "run forward") rather than simply memorizing specific motor commands.

5. Limitations and Future Work

While efficient, CB-IRL currently faces a computational bottleneck: the agent must compare its state to every expert state in the Case Base at each timestep, leading to linear complexity growth relative to the dataset size. Additionally, the current framework is optimized for "state-reaching" tasks (reaching a goal) rather than "state-keeping" tasks (such as balancing in Cart-Pole).

Future research may explore the task-agnostic nature of the Equality Net. If the network learns a general sense of temporal coherence within a specific domain, it could potentially be transferred across different tasks within the same environment, further reducing the need for new expert data.

Summary updated

BibTeX
@inproceedings{nulein2022case,
  author = {Jonas Nüßlein and Steffen Illium and Robert Müller and Thomas Gabor and Claudia Linnhoff-Popien},
  title = {{Case-Based Inverse Reinforcement Learning Using Temporal Coherence}},
  booktitle = {ICCBR},
  year = {2022},
  doi = {10.1007/978-3-031-14923-8\_20},
}
Figure