A policy trained to copy expert demonstrations often learns the wrong reason for each action. A driving policy can learn that "the car is stopped, therefore brake," and then sit at a green light forever. A locomotion policy given its previous action can learn simply to repeat it. This failure is called causal confusion, and it is especially damaging because it compounds. The confused policy drifts into states the expert never visited, and its errors grow with the length of the episode. The best existing fix, from Causal Confusion in Imitation Learning, trains one policy that takes a mask saying which input features it may use. It then picks the right mask either by asking the expert for labels or by running each masked policy and measuring the task reward. Imitation learning is used precisely when neither of those is cheap. Experts are expensive to query, and the reward is usually unknown.
We propose to choose the causal mask using only signals that imitation learning already has: the expert's demonstrations and the agent's own short rollouts. The key observation is that when the agent acts in the environment, it intervenes on the features its own actions cause, such as the previous-action indicator or its own speed. A policy that relies on such a shortcut produces rollouts whose state distribution visibly departs from the expert's. We therefore score each candidate mask by a closed-form kernel distance between the states its rollouts visit and the expert's states. This distance can be accumulated as a streaming running mean in constant memory per mask. We treat mask selection as a bandit, choosing which mask to roll out next with confidence bounds from running variance accumulators rather than from model ensembles. Finally, we refine the selected policy with a non-adversarial "value flow" objective that removes the remaining drift. The whole loop uses one network, no reward, and no expert queries.
Core Claim
Behavior cloning, the plain supervised copying of expert actions, fails on confounded inputs, and regularizers such as dropout do not repair it (Causal Confusion in Imitation Learning). Our claim is that, for confounders caused by the agent itself, reward-free rollouts supply enough interventional signal to identify the causal mask. The specific signal is the agent's state occupancy (how often its rollouts visit each state) compared against the expert's. We further claim this can be done with a fixed, small memory budget.
To make this precise, let $G$ be a binary mask over the $d$ input features and $\pi_G$ the shared mask-conditioned policy restricted to those features. Let $\phi$ be a random Fourier feature map, a fixed random projection whose inner products approximate a Gaussian kernel. Let $\mathcal{D}_E$ be the expert states and $\rho_{\pi_G}$ the distribution of states visited by $\pi_G$ in rollouts of length $L$. The score of a mask is
$$\mathrm{score}(G) = \big\| \mathbb{E}_{s \sim \rho_{\pi_G}}[\phi(s)] - \mathbb{E}_{s \sim \mathcal{D}_E}[\phi(s)] \big\|^2,$$
which is the squared maximum mean discrepancy between rollout states and expert states.
The expert mean is computed once. The rollout mean for each mask is a running average, and so is its per-coordinate variance. Memory is therefore $O(m)$ per mask, where $m$ is the number of random features, and no discriminator network is stored. A mask is selected by a lower-confidence-bound rule on the score, which concentrates rollouts on promising masks.
The design borrows two ideas, each for a different job:
- Distance without an adversary. The kernel distance comes from Maximum Mean Discrepancy Imitation Learning. We use it only to rank masks, not to train the policy.
- Confidence from running statistics. The idea of building confidence bounds from compact running statistics, instead of stored models, comes from the memory-efficient RL line. Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decomposition matches model-based exploration guarantees using $O(SAH)$ accumulators instead of an $O(S^2A)$ transition model. We do not inherit its regret guarantee, which is tabular. We inherit the design principle that variance can be tracked on the fly.
Research Program
Stage 1: mask-conditioned cloning. We train one mask-conditioned policy network by behavior cloning with randomly sampled masks, following the graph-parameterized policy of Causal Confusion in Imitation Learning. The confounded benchmarks have a small number of state factors, so we can enumerate masks or sample a few dozen. For image inputs, we work over latent factors as that paper does. The exponential growth of the mask space is a real limitation. We only partly address it, by having the bandit spend rollouts unevenly across masks.
Stage 2: bandit selection over masks. We run the bandit over masks using truncated rollouts of length $L$. The truncation is motivated by A Few Teacher Steps Go a Long Way, which shows that short partial rollouts recover much of the benefit of full on-policy data at a fraction of the cost. We will measure how short $L$ can be before the mask ranking breaks.
Stage 3: value-flow refinement. We fine-tune the selected masked policy with Dual Q-DM from Non-Adversarial Imitation Learning Provably Free of Compounding Errors. Dual Q-DM uses inequality Bellman constraints (consistency conditions on the value function) to push value from expert states back toward unvisited states, which gives error that grows linearly with horizon without adversarial training. That paper only tests unconfounded tasks, and the research file raises an open concern that value flow may carry shortcuts into unvisited states. Running Dual Q-DM on the full, confounded input is therefore both a baseline and a scientific question.
Analysis. We will examine where reward-free selection should fail, which defines its scope. When the confounder is hidden from the learner but seen by the expert, as studied in Sequential Causal Imitation Learning with Unobserved Confounders, the agent's own actions do not intervene on it. In that case we expect occupancy matching to be uninformative, and we will report this regime rather than hide it.
Tabular diagnostic. We will build a confounded variant of the Reset Cliff MDP by appending a previous-action indicator. This lets us check, with exact occupancies, when the score's minimizer coincides with the causal mask.
Baselines
Do nothing. The naive reference is behavior cloning on the confounded input, which fails completely on confounded MountainCar and Hopper according to Causal Confusion in Imitation Learning. Alongside it we run the same network with dropout, which that paper also reports failing on confounded Pong and UpNDown. The oracle ceiling is behavior cloning on the true unconfounded state.
Strongest existing method. This is de Haan et al.'s graph-parameterized policy with its two selection modes. Policy-execution mode scores masks by environment reward. Expert-query mode queries the expert at states where candidate masks disagree. The file contains no reported numeric result on Lacuna for either mode on confounded MountainCar, Hopper, or Atari. The paper states only that the modes "recovered near-expert performance within just a few dozen episodes or queries," and that on Pong and UpNDown they matched the unconfounded model.
We will rerun both modes with:
- the same mask-conditioned network and the same demonstrations;
- the same budget of environment steps;
- an equal hyperparameter search budget.
Policy-execution mode gets privileged reward access, so it is an upper reference rather than a fair competitor. Beating it is not required. Our method succeeds if it matches the unconfounded-state performance that policy-execution mode reaches, within its confidence interval, using no reward and no queries, at comparable environment steps.
Value flow on the full input. Dual Q-DM is trained on the full confounded input with the same number of environment steps. It is the strongest reward-free, query-free competitor. The Dual Q-DM paper reports no numeric returns per environment in the file.
Q-ensemble prioritized sampling. We adapt the method of Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning? as the memory-heavy reference. That paper reports convergence in roughly a quarter of the gradient steps of uniform sampling on Traffic-World, and about 40% higher final reward on confounded Enduro. We compare peak GPU memory and number of hosted networks as well as return.
Ablations. Two ablations isolate the key ingredient:
- Ranking signal. Replace the bandit over masks with uniform rollout allocation. This tests whether the running-variance confidence bounds matter.
- Score function. Replace the streaming kernel score with a learned discriminator. This tests whether the cheap closed-form distance loses ranking accuracy.
Why not just…?
Why not just run value flow or GAIL on the full input, since online interaction already intervenes? This is the most serious threat. If occupancy matching identifies the shortcut, it is possible that matching occupancy during training, as Dual Q-DM or Generative Adversarial Imitation Learning do, already pushes the network away from it. Our reason to doubt this is that the network is never forbidden from reading the shortcut. Gradient training can settle on a solution that is partly confused and still roughly matches occupancy, and the Gap 2 concern is that Bellman constraints spread value along whatever features the network uses. We do not assume this. The full-input Dual Q-DM baseline settles it directly. If that baseline reaches unconfounded-state performance on confounded Hopper and MountainCar, we will report that value flow alone resolves self-caused confusion and drop the mask machinery. That would itself be a useful negative result for this proposal.
Why not just remove the suspicious feature? That requires already knowing which feature is spurious. That is the oracle row in our table, not a method. On visual or factor-rich inputs nobody hands us the list. Even ReCCoVER: Detecting Causal Confusion for Explainable Reinforcement Learning needs a human to define the problematic subspaces after flagging them.
Why not just query the expert, DAgger-style? Interactive querying works when an expert is available, but it is costly. ThriftyDAgger exists precisely to ration such queries. de Haan's expert-query mode already needs far fewer queries than DAgger, and our method needs none. The comparison is reported as performance at zero queries versus the query count the expert mode needs to reach the same return.
Why not just add more demonstrations? More data from the same confounded expert keeps the same correlation. RoCoDA reports vanilla cloning on Libero-Object falling to 22.3% success under unseen distractors, against 99.3% for counterfactual augmentation. In our experiments, we will scale demonstrations fourfold for the cloning baseline.
Proposed Experiments
Main benchmarks. These are confounded MountainCar, confounded Hopper, and confounded Pong, Enduro and UpNDown from Causal Confusion in Imitation Learning, plus our confounded Reset Cliff variant. To test whether we harm clean tasks, we also run the unconfounded MuJoCo suite used by Dual Q-DM. Reward is used only for evaluation.
Metrics. We report:
- normalized return relative to the unconfounded-state oracle;
- environment steps to reach 90% of the oracle;
- peak GPU memory and number of networks hosted;
- wall-clock time;
- on Reset Cliff, imitation gap as a function of horizon.
Decisive check 1: is the signal there at all? On every confounded task, the causal mask must achieve the lowest score among enumerated masks, given enough rollouts. If it does not, the core claim is false.
Decisive check 2: does value flow alone suffice? This is the full-input Dual Q-DM comparison described above.
Decisive check 3: are short rollouts enough? We sweep rollout length $L$ and record the shortest length at which the mask ranking stays correct. This measures how much environment cost truncation actually saves.
Scope check. We deliberately run a variant where the confounder is visible to the expert but hidden from the learner, where we predict failure. This marks the boundary of reward-free selection.
References
- Causal Confusion in Imitation Learning
- Maximum Mean Discrepancy Imitation Learning
- Almost Optimal Model-Free Reinforcement Learningvia Reference-Advantage Decomposition
- A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Training
- Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism
- Sequential Causal Imitation Learning with Unobserved Confounders
- Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?
- Model-Free Imitation Learning with Policy Optimization
- ReCCoVER: Detecting Causal Confusion for Explainable Reinforcement Learning
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations