Integrating Action Prefix Prediction and Alignment Across Modalities for Multimodal Reinforcement Learning

Sijie Xu, Francisco CruzarXiv 2026 · arXiv

TL;DR

The authors introduce Improved Predictive Representation (IPR), a framework that enhances multimodal reinforcement learning by replacing slow, error-prone recursive latent rollouts with parallelized action-prefix prediction. By combining this with a more sophisticated cross-modal alignment strategy (CoMM) at the observation level, IPR achieves up to a 10% improvement in returns on standard benchmarks and a 1.15x training speedup.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often struggle to integrate information from multiple sensors—like RGB cameras, depth maps, and position vectors—especially when interaction data is limited. A recent approach called Multi-Modal Joint Predictive Representation (MJPR) tried to solve this by predicting future latent states and aligning different modalities.

However, MJPR has two major bottlenecks:

  1. Recursive Rollouts: It predicts the future step-by-step ($z_t \rightarrow z_{t+1} \rightarrow z_{t+2}$). If the first prediction is slightly off, the error compounds, leading to poor long-term representations. Furthermore, these steps must be computed sequentially, which is slow.
  2. Simplistic Alignment: Its alignment strategy focuses mostly on what information is shared between sensors, potentially ignoring unique, task-relevant features present in only one modality.

Methods

IPR introduces two key architectural shifts to the MJPR framework, as illustrated in the high-level overview below:

Figure 1: High-level overview of IPR using visual and position observations as an example. The encoders provide representations to SAC and the two auxiliary branches. Action-prefix prediction provides temporal supervision, while CoMM style alignment relates complete and modality-retaining views at the anchor observation. Superscripts v and p denote visual and position representations.

1. Action-Prefix Latent Prediction

Instead of a recursive chain, IPR uses an action-prefix approach. It takes a single "anchor" observation and a sequence of actions (the prefix) to predict a future state directly. For example, to predict the state three steps ahead, it uses the anchor and the sequence $(a_t, a_{t+1}, a_{t+2})$ simultaneously. This allows the model to compute multiple future horizons in parallel and avoids the "compounding error" problem of recursive models.

2. Anchor-Level CoMM Alignment

Rather than aligning modalities during the predicted rollout, IPR applies Contrastive Multimodal learning (CoMM) at the initial anchor observation. It creates "complete views" (all sensors) and "modality-retaining views" (where all but one sensor are masked). By contrasting these, the model learns to capture redundant, unique, and synergistic information across all inputs.

The detailed architecture of these branches is shown here:

Figure 2: Detailed view of the prediction and alignment branches. The prediction branch uses a causal Transformer to process action prefixes.

Datasets

The researchers evaluated IPR across two distinct environments:

  • DeepMind Control Suite (DMC): Five continuous control tasks (Catch, Run, Spin, Swing-up, and Walk) using two sensor configurations: RGB–Depth (where images are masked to force reliance on both) and RGB–Position.
  • Webots Navigation: A simulated E-puck robot navigation task requiring the agent to find targets while avoiding obstacles, tested with and without LiDAR data.
Figure 3: Example RGB camera observations from the five DeepMind Control Suite tasks. The added arrows show the main motion directions and are not part of the agent observations.

Key Insights and Ideas

  • Parallelism is a Feature, Not Just a Speedup: While parallelizing the rollout reduces training wall time by ~13%, its primary benefit in this context is stability. By grounding every prediction in a real, observed anchor rather than a previously predicted (and potentially noisy) latent state, the representation stays much closer to reality.
  • Alignment is Better at the Source: The authors found that applying complex augmentations (like random cropping) to predicted latent representations actually hurt performance. By moving the alignment objective to the observed anchor, they could use standard image augmentation techniques effectively.
  • Context Matters for Cropping: The paper compared two cropping intensities: IPR-S (strong, 10% area) and IPR-H (higher retention, 50% area). They found that "more is not always better"—the ideal crop strength depends heavily on the specific task and how much visual context is required to solve it.

Key Results

  • Performance Boost: In the DMC RGB–Depth setting, IPR-H achieved an average return 10.0% higher than the MJPR baseline. In the RGB–Position setting, it was 3.4% higher.
  • Efficiency: Replacing recursive rollouts with action-prefix prediction resulted in a 1.15x speedup in total training wall time.
  • Robustness in Webots: When LiDAR (direct range data) was removed, IPR significantly outperformed the baseline. This suggests that IPR’s predictive and alignment signals are most valuable when the agent has to "infer" spatial relationships from less direct sensors like cameras.
Figure 7: Learning curves in Webots. Note how IPR (orange/green) performs better than the baseline (blue) when LiDAR is removed in plot (b).

Limitations and Future Work

  • Statistical Significance: The authors noted substantial variation across different random seeds, meaning that while the trends are positive, more extensive testing is needed to establish absolute superiority.
  • Sensor Redundancy: In the Webots environment with LiDAR, the baseline actually performed better. The authors hypothesize that when sensors provide highly redundant, high-quality information, the extra complexity of IPR’s auxiliary losses might act as a distraction rather than a help.
  • Real-World Transfer: The study was conducted entirely in simulation; testing how these representations hold up against the noise and calibration issues of real-world robotic sensors remains a future goal.

Summary updated

Abstract

Learning representations from multimodal observations is difficult in reinforcement learning with limited interaction data. Multimodal Joint Predictive Representation combines multimodal latent prediction with contrastive learning across modalities, but its recursive rollout depends on earlier predictions and its pairwise alignment focuses on correspondence between modalities. We propose Improved Predictive Representation, which combines prediction from action prefixes with alignment at the observed anchor based on Contrastive Multimodal learning. At each horizon, it predicts future representations directly from the observed anchor and the corresponding sequence of actions. We evaluate two crop configurations: IPR-S with a minimum retained crop area of 10%, and IPR-H with 50%. Across five DeepMind Control Suite tasks, IPR-H achieves an average return across tasks that is 10.0% higher than the baseline with visual and depth images and 3.4% higher with visual images and position vectors. The relative performance of the two configurations varies across tasks. In Webots, both IPR configurations achieve higher mean final returns than the baseline without LiDAR, but not when LiDAR is available.
BibTeX
@misc{xu2026integrating,
  author = {Sijie Xu and Francisco Cruz},
  title = {{Integrating Action Prefix Prediction and Alignment Across Modalities for Multimodal Reinforcement Learning}},
  year = {2026},
  eprint = {2611.13782},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13782},
}
Figure