Causal Transfer for Imitation Learning and Decision Making under Sensor-Shift

Jalal Etesami, Philipp Geiger

Abstract

Learning from demonstrations (LfD) is an efficient paradigm to train AI agents. But major issues arise when there are differences between (a) the demonstrator's own sensory input, (b) our sensors that observe the demonstrator and (c) the sensory input of the agent we train. In this paper, we propose a causal model-based framework for transfer learning under such “sensor-shifts”, for two common LfD tasks: (1) inferring the effect of the demonstrator's actions and (2) imitation learning. First we rigorously analyze, on the population-level, to what extent the relevant underlying mechanisms (the action effects and the demonstrator policy) can be identified and transferred from the available observations together with prior knowledge of sensor characteristics. And we device an algorithm to infer these mechanisms. Then we introduce several proxy methods which are easier to calculate, estimate from finite data and interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. We validate our two main methods on simulated and semi-real world data.

Introduction: The Sensor-Shift Problem

In Learning from Demonstrations (LfD), the goal is to train an AI agent by observing an expert. However, a significant practical hurdle is sensor-shift, where the sensory inputs of the demonstrator, the spectator (recording equipment), and the final agent do not align.

A typical sensor-shift scenario: a drone (spectator) observes a highway scene but may miss critical cues like indicator lights visible to the demonstrator.

Consider a highway driving scenario: a driver (demonstrator) monitors mirrors and indicator lights. A drone (spectator) recording from above might miss these subtle cues. If an AI agent naively mimics the drone footage, it may suffer from "causal confusion"—for instance, failing to understand that a lead car's blinker caused the demonstrator to slow down. Such misunderstandings can lead to dangerous deployment failures.

Methodology

We propose a framework based on Causal Models (Directed Acyclic Graphs) to bridge these sensory domains by identifying invariant mechanisms. Our approach addresses two primary tasks:

  1. Action-Effect Transfer: Inferring how actions (e.g., braking) affect the environment in the target domain.
  2. Imitation Transfer: Learning a policy that mimics the expert's behavior despite differing sensor inputs.

Algorithms and Proxies

We developed two core algorithms: Algorithm 1 for discrete variables, which uses Singular Value Decomposition (SVD) to identify hidden states, and Algorithm 2 for linear continuous data, which utilizes covariance matrices and Schur complements to estimate true action effects. To address computational complexity and finite data constraints, we also introduced "average-based proxies." These simplified methods include theoretical bounds (using Kullback-Leibler divergence) to ensure they remain close to the ground-truth causal dynamics.

Experimental Evaluation

We validated our framework using a simulated driving environment and the highD dataset, which contains real-world aerial drone recordings of German highways. This provided a "semi-real" testbed where car positions are known, but causal factors like driver intent (signaled by blinkers) are missing from the spectator's view.

Performance of the proposed methods. Left: Mean squared test error for action-effect transfer comparing the exact linear method vs. proxies. Right: Convergence of policies in imitation learning.

Key Results

  • Action-Effect Transfer: In highway drone experiments, the exact linear method (Algorithm 2) significantly outperformed average-based baselines as the dataset size increased, successfully recovering causal dynamics that naive models missed.
  • Imitation Learning: In simulations where the imitator lacked the demonstrator's full view, our causal method (Algorithm 1) converged to the true expert policy. Standard imitation learning approaches failed because they could not account for missing causal information (e.g., the hidden indicator light).

Discussion and Conclusion

The core insight of this work is that while sensors change, the underlying "physics" of the environment and the expert's internal logic often remain invariant. Causal models allow us to extract these invariant components from noisy, shifted observations.

While the current framework requires some prior knowledge of sensor characteristics and faces computational challenges in high-dimensional discrete states, it provides a rigorous foundation for safe transfer learning. Future work will explore non-linear relationships and scenarios where the causal structure itself is only partially known.

Summary updated

Abstract

Learning from demonstrations (LfD) is an efficient paradigm to train AI agents. But major issues arise when there are differences between (a) the demonstrator's own sensory input, (b) our sensors that observe the demonstrator and (c) the sensory input of the agent we train.In this paper, we propose a causal model-based framework for transfer learning under such “sensor-shifts”, for two common LfD tasks: (1) inferring the effect of the demonstrator's actions and (2) imitation learning. First we rigorously analyze, on the population-level, to what extent the relevant underlying mechanisms (the action effects and the demonstrator policy) can be identified and transferred from the available observations together with prior knowledge of sensor characteristics. And we device an algorithm to infer these mechanisms. Then we introduce several proxy methods which are easier to calculate, estimate from finite data and interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. We validate our two main methods on simulated and semi-real world data.
BibTeX
@inproceedings{etesami2020causal,
  author = {Jalal Etesami and Philipp Geiger},
  title = {{Causal Transfer for Imitation Learning and Decision Making under Sensor-Shift}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=wnvltYFuYv},
}
Figure