Agents that learn from camera images get distracted by things they cannot control: background video, other cars, other robots. One reliable way to ignore such distractions is inverse dynamics, meaning training an image encoder by asking it to guess which action the agent took from what the world looked like before and after. Anything that moves regardless of the agent's actions carries no information about that action, so the encoder learns to drop it. The strongest version of this idea looks several steps ahead. It predicts the action at time $t$ from the frame at $t$ and a frame $k$ steps later, which blocks the shortcut of reading off a one-frame flicker. Current methods pick $k$ by hand or sample it at random, and each prediction uses a single future frame. We propose to replace that single future frame with a learned, linear-time summary of the entire future beyond a fixed gap. A reverse-time selective state-space scan, a recurrent layer whose write strength depends on its input, reads the future frames. It can then decide per trajectory which future frames reveal what the agent did at time $t$.
This matters because the right horizon is not a constant. It depends on actuator lag, on how long an action's effect stays distinguishable from noise, and on how persistent the data-collecting policy's actions are. Offline datasets mix all three. A single pooled future summary that is computed in one backward pass also scales to trajectories of thousands of steps, where comparing every pair of time steps or using full attention over the future becomes costly. The claim we test is that the new ingredient, gap-masked, input-dependent pooling of future evidence, improves exogenous-noise filtering when effects are delayed or horizons vary. The claim is not that a newer backbone helps.
Core Claim
Let $z_i = \phi(x_i)$ be the per-frame encoding of observation $x_i$. A reverse selective scan computes, for every step $j$, a summary of all frames from $j$ to the end of the window $T$:
$$s_j = \bar{A}(z_j)\, s_{j+1} + \bar{B}(z_j)\, z_j, \qquad s_{T+1} = 0,$$
In this scan, $\bar{A}(\cdot)$ and $\bar{B}(\cdot)$ are discretized state-space matrices whose step size depends on the input, as in Mamba-style selective models. The encoder is trained with
$$\mathcal{L}(\phi, g) = \mathbb{E}_t\left[\mathrm{CE}\big(g(z_t, s_{t+m}),\, a_t\big)\right],$$
Here $g$ is a small action classifier (or regressor, for continuous actions) and $m \ge 2$ is a gap that the summary cannot see across.
The gap removes the lazy single-step cue: frames $t+1$ through $t+m-1$ never enter the summary. The input-dependent step size lets the scan keep future frames where the action's consequence is legible and flush frames where it is washed out, which amounts to a learned horizon. A single backward pass computes every $s_{t+m}$, so the cost is linear in $T$. Actions are never fed as inputs, which avoids the "copycat" shortcut that LRAM handled by stripping past actions.
Our conjecture is that, under the exogenous block MDP assumptions of Efroni et al., conditioning on any set of future observations beyond the gap preserves the key property. The Bayes-optimal predictor needs only endogenous state, and exogenous state stays conditionally irrelevant. This requires the behavior policy to ignore exogenous state, which is the same assumption ACRO makes. We will check the argument formally for the discrete case and empirically everywhere else.
Relation to Prior Work
Three works motivate the proposal by showing that multi-step inverse objectives filter correlated noise where single-step ones do not. Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics formalizes endogenous/exogenous factorization and uses path elimination over action sequences. ACRO (Agent-Controller Representations, Principled Offline RL in the Presence of Rich Exogenous Information) trains an encoder to predict $a_t$ from a pair $(x_t, x_{t+k})$ with $k$ sampled, and shows this restores Bellman completeness in visual offline RL. Both treat the horizon as fixed or randomly sampled and use one endpoint frame per prediction. We keep their objective's logic but replace the endpoint with a pooled, learned-horizon future, and we directly target the horizon-selection weakness that the file lists as open.
Single-step approaches constrain the design by showing what the gap must prevent. ICM, RILIR, and the action-conditioned branch of Iso-Dream all use adjacent-frame inverse losses. Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation shows that such losses deconfound hidden goals. None of them pools future evidence or controls the shortcut through near-term frames.
The linear-complexity work supplies the mechanism. VMamba provides selective scans with linear cost. LRAM shows that recurrent linear models keep flat memory over 1,600-step robot contexts, but it trains by behavior cloning with no distractor filtering. MambaOut is the key constraint: on short sequences gated convolutions beat Mamba. This is why we do not claim a spatial backbone swap helps at 84×84 resolution, and why we put the state-space layer on the temporal axis, where sequences are long. Iso-Dream also separates controllable from uncontrollable dynamics, but it does so with a generative world model and a one-step inverse regularizer. Our method is encoder-only, avoids reconstruction, and is multi-step with a learned horizon.
Research Program
We will build three things. The first is the gap-masked reverse scan objective on top of the ACRO encoder and training pipeline, so that the per-frame encoder and the downstream offline RL learner match ACRO exactly. The second is a family of controlled stress variants of the exogenous-distractor suite that vary the parts of the problem a fixed $k$ gets wrong. One variant adds actuator delay of $d$ steps, where $d$ differs across episodes and is unknown to the learner. A second adds action persistence in the behavior policy, with repeat probability swept from 0 to 0.9. A third adds exogenous video whose correlation timescale is swept from a few frames to hundreds. The third is an analysis of what the scan learns. We will read out the input-dependent step sizes to obtain an effective horizon per trajectory, check whether it tracks the injected delay $d$, and probe whether exogenous variables such as the background agents' positions in the 9-agent grid can be decoded linearly from $z$.
The theoretical piece is modest. For the discrete, near-deterministic setting of Efroni et al., we will check whether replacing the single endpoint with the future set beyond the gap preserves identifiability of endogenous state. We will also characterize when it fails, for example when late future frames depend on later actions in ways that correlate with exogenous state through the policy.
Baselines
The naive reference point is the offline RL learner used in the ACRO experiments, trained directly from pixels with augmentation and no representation pretraining. Following the ACRO papers, this is DrQ-v2- or CQL-style training. It shows how much any inverse objective buys. Alongside it we run a frozen random encoder to calibrate the floor.
The strongest existing method is ACRO itself. It uses the identical per-frame convolutional encoder, sampled $k$ as in the paper, and the same downstream learner. We will tune its $k$ range within the same budget as our method: an equal number of hyperparameter configurations, the same dataset, the same number of gradient steps, and matched encoder parameter count. The file reports only qualitatively that ACRO "maintained high performance, often matching the scores achieved on 'clean' data" on Hard-Exo, while DRIML, CURL, and DrQ-v2 collapsed to near-zero returns. We found no reported numeric result on Lacuna for ACRO on Hard-Exo or the 9-agent grid, so we will reproduce ACRO before comparing against it. Beating it means a statistically significant gain in normalized return on the delay and persistence stress variants, measured over at least five seeds, with no loss on standard Hard-Exo. It also means lower linear decodability of exogenous variables from $z$.
A stronger version of ACRO averages the inverse loss over all pairs $(t, t+k)$ with $k \ge m$ in each window. This tests whether our gain comes merely from using more future frames.
We also run a single-step inverse baseline in the style of RILIR and Iso-Dream's inverse head. It anchors the claim that a multi-step objective matters on correlated noise. For the online distracting-video setting, the file reports Iso-Dream at 911 against DreamerV2 at 655 on Walker Walk with natural video backgrounds (Iso-Dream). That is an online model-based result, so we cite it only as context and not as a matched comparison.
Three ablations isolate the key ingredient. The first removes the gap ($m=1$) to test whether the gap blocks shortcuts. The second replaces the selective scan with a non-selective, fixed-decay scan, which leaves input-independent exponential pooling. The third replaces it with gap-masked attention over the future frames, which is quadratic but equally expressive, to separate selectivity from compute.
Why not just…?
Why not just use ACRO with a well-tuned $k$ or with all pairs? On datasets with a single known delay and non-persistent actions, this may match us, and we expect it to. The hypothesis is that any fixed or averaged $k$ is wrong somewhere when delays vary across episodes or actions persist. Averaging pairwise losses also never pools evidence across futures. The all-pairs baseline and the delay-sweep experiment settle this directly. If all-pairs ACRO matches us across the delay and persistence sweeps, the adaptive-horizon claim is false, and we would report the method as an efficient implementation of all-pairs ACRO and nothing more.
Why not just pool the future with a transformer? For windows of a few hundred steps, gap-masked attention is affordable and may perform as well. This is a real threat to any claim that selectivity matters, and the attention ablation measures it. What linear cost buys is behavior at long windows (1,000 to 2,000 steps, matching the context lengths where LRAM reports Transformer throughput degrading and running out of memory) and streaming use. We will report training throughput and memory against window length for both variants. If attention wins at every feasible length, we will say so.
Why not just swap ACRO's CNN for a VMamba encoder? That is the obvious intersection, and MambaOut argues against it at typical visual RL resolutions. There, gated depthwise convolutions reach 84.1% ImageNet top-1 against 83.7% for LocalVMamba-S, at 79% of the compute, with a turning point only near thousands of tokens. We include an encoder-swap arm (CNN, gated-conv, VMamba) crossed with our objective at 84×84 and at an enlarged, high-resolution 9-agent grid. We predict the swap does little at low resolution. Our temporal claim does not depend on the swap.
Why not just use a decoupled world model like Iso-Dream? Reconstruction-based models spend capacity on distractors. The ACRO papers argue that retaining exogenous features breaks Bellman completeness offline. Iso-Dream's controllable branch relies on the single-step inverse loss that correlated noise defeats. We treat it as context rather than as a matched baseline because it is an online, generative method.
Proposed Experiments
The primary testbed is the v-d4rl-based suite from the ACRO papers: Easy-, Medium-, and Hard-Exo on Cheetah, Walker, and Humanoid, plus the 9-agent grid. We add our delay, persistence, and correlation-timescale variants. Metrics are normalized return, linear-probe accuracy for endogenous state (joint positions and velocities) and for exogenous state (background-agent positions or video frame index), and training throughput and memory against window length.
The secondary testbed is the visual grid world with moving ellipses and the noisy combination lock from Efroni et al.. There we measure state-decoding accuracy and the largest solved horizon. PPE's reported solved horizon of $H=40$, against baselines failing at $H=5$ to $10$, is the reference, and we plug our encoder into the same exploration loop.
The decisive checks are three. First, on the delay sweep, the scan's learned effective horizon should rise with the injected delay $d$, and its return should stay flat where fixed-$k$ ACRO degrades for delays outside its sampled range. Second, removing the gap should make exogenous decodability climb on correlated-video tasks, showing that the gap and not merely pooling is what blocks spurious cues. Third, the non-selective scan should fall between ACRO and the full method on variable-delay data. If it matches the full method, selectivity is unnecessary and the contribution reduces to gap-masked pooling, which we would report as the finding.
References
- A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Tasks
- Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics
- Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information
- Principled Offline RL in the Presence of Rich Exogenous Information
- Curiosity-driven Exploration by Self-supervised Prediction
- Robust Visual Imitation Learning with Inverse Dynamics Representations
- Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models
- Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation
- VMamba: Visual State Space Model
- MambaOut: Do We Really Need Mamba for Vision?