Causal Transfer for Imitation Learning and Decision Making under Sensor-shift

Jalal Etesami, Philipp Geiger

Abstract

Learning from demonstrations (LfD) is an efficient paradigm to train AI agents. But major issues arise when there are differences between (a) the demonstrator's own sensory input, (b) our sensors that observe the demonstrator and (c) the sensory input of the agent we train. In this paper, we propose a causal model-based framework for transfer learning under such "sensor-shifts", for two common LfD tasks: (1) inferring the effect of the demonstrator's actions and (2) imitation learning. First we rigorously analyze, on the population-level, to what extent the relevant underlying mechanisms (the action effects and the demonstrator policy) can be identified and transferred from the available observations together with prior knowledge of sensor characteristics. And we device an algorithm to infer these mechanisms. Then we introduce several proxy methods which are easier to calculate, estimate from finite data and interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. We validate our two main methods on simulated and semi-real world data.

Introduction: The Sensor-Shift Problem

In Learning from Demonstrations (LfD), a common assumption is that the student agent perceives the environment exactly as the demonstrator did. However, in real-world applications, this is rarely the case. Consider a scenario where a drone records highway traffic to train autonomous vehicles. The human driver (demonstrator) sees the road and their dashboard; the drone (spectator) sees the scene from above but may miss small details like indicator lights; and the target AI agent has its own unique sensor suite.

The sensor-shift problem illustrated: A spectator (e.g., a drone) might miss the lead car's indicator light, leading to causal confusion.

This discrepancy is termed "sensor-shift." Ignoring it leads to the learning of "spurious correlations." For instance, if a drone fails to detect a lead car's indicator, it might incorrectly conclude that the demonstrator's braking caused the lead car to change lanes, rather than recognizing both were responding to a signal invisible to the drone. This leads to "causal confusion," where the agent fails to understand the true dynamics of the environment.

Methodology: Causal DAGs and Invariant Mechanisms

The authors address sensor-shift as a transfer learning problem using Causal Directed Acyclic Graphs (DAGs). The fundamental premise is that while sensors vary, the underlying "physics"—the causal mechanism of how actions affect outcomes—remains invariant across domains. By modeling these differences as causal shifts, the framework allows for the recovery of true action-effects and expert policies even when crucial information is missing or noisy in the training data.

Key technical contributions include:

  • Identification Analysis: A rigorous definition of when it is mathematically possible to recover mechanisms from shifted data.
  • Algorithm 1 (Discrete Case): A method to find the set of all possible solutions for discrete variables using techniques inspired by Singular Value Decomposition (SVD).
  • Algorithm 2 (Linear Continuous Case): A sample-level algorithm that uses covariance matrices and Schur complements to estimate transfer functions.
  • Average-based Proxies: Computationally efficient approximations with proven theoretical bounds on how close they are to the ground truth.

Experimental Validation

The framework was validated using two primary data sources:

  1. The "highD" Dataset: A large-scale real-world dataset of naturalistic vehicle trajectories recorded by drones over German highways. This was used to test action-effect transfer, specifically predicting how one car's acceleration affects another.
  2. Simulated Driving Scene: A custom simulation designed to replicate the "indicator light" problem, where a hidden coordination signal exists that the imitator cannot see directly.

Key Results

The "Exact Linear Method" (Algorithm 2) demonstrated superior performance in action-effect transfer. While exhibiting higher variance with small datasets, it converged significantly closer to the true causal effect as training data increased compared to naive baselines.

Mean squared test error for action-effect transfer. The exact linear method converges toward the performance of a model with full observation as sample size increases.

In imitation learning simulations, the causal imitator successfully learned a policy that avoided crashes. In contrast, the "naive" imitator, which ignored sensor-shift, failed because it could not account for the hidden common cause (the indicator light), leading to unpredictable and dangerous decisions.

Discussion and Future Work

While the proposed framework provides a robust solution for sensor-shift, it currently assumes a known model of sensor characteristics (e.g., $P(Y_S|X)$). In many real-world scenarios, these might be unknown and would need to be learned simultaneously. Furthermore, while the paper covers discrete and linear continuous cases, the general non-linear continuous case remains a significant challenge due to the complexity of the underlying mathematics. Future directions include extending this work to complex temporal models like Reinforcement Learning and exploring ways to relax the requirement for prior knowledge of sensor noise.

Summary updated

Abstract

Learning from demonstrations (LfD) is an efficient paradigm to train AI agents. But major issues arise when there are differences between (a) the demonstrator's own sensory input, (b) our sensors that observe the demonstrator and (c) the sensory input of the agent we train. In this paper, we propose a causal model-based framework for transfer learning under such "sensor-shifts", for two common LfD tasks: (1) inferring the effect of the demonstrator's actions and (2) imitation learning. First we rigorously analyze, on the population-level, to what extent the relevant underlying mechanisms (the action effects and the demonstrator policy) can be identified and transferred from the available observations together with prior knowledge of sensor characteristics. And we device an algorithm to infer these mechanisms. Then we introduce several proxy methods which are easier to calculate, estimate from finite data and interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. We validate our two main methods on simulated and semi-real world data.
BibTeX
@inproceedings{etesami2020causal,
  author = {Jalal Etesami and Philipp Geiger},
  title = {{Causal Transfer for Imitation Learning and Decision Making under Sensor-shift}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=wnvltYFuYv},
}
Figure