CAR-DESPOT: Causally-Informed Online POMDP Planning for Robots in Confounded Environments

Ricardo Cannizzaro, Lars Kunze

TL;DR

CAR-DESPOT introduces a causally-informed online POMDP planner that eliminates policy errors caused by unobserved environmental confounders. By integrating Structural Causal Models (SCMs) and interventional inference ($do$-calculus) into the state-of-the-art AR-DESPOT framework, the planner successfully separates spurious correlations from true environmental physics, leading to significantly safer and more optimal robotic policies.

What Problem Does This Paper Address?

Robots operating in the real world rely on Partially Observable Markov Decision Processes (POMDPs) to plan actions under uncertainty. However, standard POMDPs lack causal semantics. They assume that the relationship between an action and its outcome can be fully captured by observational data: $P(S'|S,A)$.

This becomes problematic in the presence of unobserved confounders—hidden variables that simultaneously influence both the robot's action selection (e.g., its reactive or reflexive behaviors) and the state transitions. For instance, consider a robot navigating near a hidden electromagnet that disrupts its compass and skews its movement. If the robot's historical data shows it frequently failing when attempting to move "UP" in that area, a standard POMDP planner will conclude that the "UP" action itself is inherently unreliable. In reality, the failure was caused by the hidden magnet, not the action. This confusion of correlation with causation is known as confounding bias, and it frequently leads standard planners to generate highly suboptimal, overly conservative, or dangerous policies.

Fig. 1: (a) GridWorldConfounded. The robot moves in the four cardinal directions in a grid to reach the goal while avoiding occupied cells. Variable sensor interference (orange cell) from an electromagnet is an unobserved confounder. (b) The SCM representation of a UCPOMDP. The agent’s reactive action selection and state updates are both condi- tionally dependent on the unobserved confounder. Causal interventions (red crosses) are taken on actions to eliminate confounding bias in transition predictions. Observed vari- ables are shown as white nodes, unobserved as grey.

Methods

The authors address this by proposing CAR-DESPOT (Causally-Informed Online POMDP Planning), which breaks down into two main phases:

  1. Offline Model Parameter Learning: The system dynamics are modeled as an SCM with Unobserved Confounding (SCM-UCPOMDP). Assuming a "privileged" training phase where the confounder can be temporarily measured (e.g., during a human-guided initial site survey), the authors use Stochastic Variational Inference (SVI) via the Pyro probabilistic programming language to learn the latent parameters of the transition function and the confounder's distribution.
  2. Online Causal Planning: During online deployment, the confounder is completely unobserved. Instead of sampling from the biased observational distribution $P(S'|S,A)$ during tree rollouts, CAR-DESPOT uses the causal $do$-operator to simulate interventions: $P(S'|S, do(A=a))$. This mathematically cuts the causal link between the unobserved confounder and the action selection, isolating the true physical effect of the action.

Datasets

The method was evaluated on a newly introduced synthetic benchmark called GridWorldConfounded.

  • The Setup: A 2D grid navigation task where a robot must reach a goal while avoiding obstacles.
  • The Confounder: An unobserved electromagnet placed at a specific cell coordinates. When the robot enters the adjacent cell, the magnet introduces a fluctuating orientation error ($U_{\text{OrientationError}}$) that skews the robot's actual movement and triggers internal, hardcoded reactive behaviors.

Key Insights and Ideas

  • The Observational vs. Interventional Shift: The core "aha moment" is visualized in the transition probabilities. As shown in Figure 2, the observational model severely misrepresents the true physics. For the "MOVE_UP" action under confounding, the observational distribution predicts a success probability of only $0.1914$, whereas the interventional distribution reveals the true physical success probability is actually $0.7229$.
  • Why Standard Planners Fail: Because standard AR-DESPOT relies on observational data, it undervalues perfectly good actions simply because they were historically correlated with bad conditions. This forces the baseline planner to take long, inefficient detours to avoid phantom risks.
Fig. 2: Interventional and observational transition probabili- ties of ground truth and learned models, under the influence of confounding, as a function of action choice. Formulating actions as causal interventions removes confounding bias errors present in in the observational transition predictions.

Key Results

  • High Learning Accuracy: The offline SVI method successfully recovered the 106 latent model parameters from 800,000 observational data points, achieving a remarkably low Kullback–Leibler (KL) divergence of $D_{KL} = 0.0021$ compared to the ground truth.
  • Superior Planning Performance: In closed-loop evaluations across 100 runs, CAR-DESPOT dramatically outperformed the baseline (OAR-DESPOT). When planning with the learned model, CAR-DESPOT achieved a mean total discounted reward of $21.93 \pm 9.21$, compared to the baseline's dismal $0.74 \pm 7.73$.
  • Strategic Optimality: CAR-DESPOT recognized that the shorter path was physically viable despite the confounder, resulting in faster goal alignment and fewer opportunities to accidentally drift into walls.

Limitations and Future Work

Despite its success, the authors highlight two major limitations:

  • State Representation Constraints: The current parameter learning method assumes transition dynamics can be cleanly decomposed into relative state changes. This works well for 2D grids but may struggle to scale to complex, continuous robotic state spaces without requiring massive datasets.
  • Computational Overhead: The online search time required to compute a policy at each step was roughly 150 seconds. This is currently too slow for real-time robotic applications and requires optimization.

Future Direction: The authors plan to transition from interventional inference to counterfactual inference. Counterfactuals would allow the robot to look at its immediate reactive action choices and retroactively infer the current hidden state of the confounder, unlocking even more adaptive behavior.

Summary updated

Abstract

Robots operating in real-world environments must reason about possible outcomes of stochastic actions and make decisions based on partial observations of the true world state. A major challenge for making accurate and robust action predictions is the problem of confounding, which if left untreated can lead to prediction errors. The partially observable Markov decision process (POMDP) is a widely-used framework to model these stochastic and partially-observable decision-making problems. However, due to a lack of explicit causal semantics, POMDP planning methods are prone to confounding bias and thus in the presence of unobserved confounders may produce underperforming policies. This paper presents a novel causally-informed extension of "anytime regularized determinized sparse partially observable tree" (AR-DESPOT), a modern anytime online POMDP planner, using causal modelling and inference to eliminate errors caused by unmeasured confounder variables. We further propose a method to learn offline the partial parameterisation of the causal model for planning, from ground truth model data. We evaluate our methods on a toy problem with an unobserved confounder and show that the learned causal model is highly accurate, while our planning method is more robust to confounding and produces overall higher performing policies than AR-DESPOT.
BibTeX
@inproceedings{cannizzaro2023car,
  author = {Ricardo Cannizzaro and Lars Kunze},
  title = {{CAR-DESPOT: Causally-Informed Online POMDP Planning for Robots in Confounded Environments}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342223},
}
Figure