SWIM: Learning What Matters from Behaviors for JEPA World Models

Jiaming Hu, Shi Bo, Yan Zheng, Tian Wang, Florian Dubost, Alejandro Mottini, Junze Liu, Arvind Srinivasan, Kai Zhong, Kun Qian, Sharon Gao, Vianne Gao, Houyu Zhang, Qingjun CuiarXiv 2026 · arXiv

TL;DR

SWIM (State Weighting with Inverse-Dynamics Model) is a two-stage framework that automatically learns which physical state features are task-relevant by training an inverse-dynamics model on offline data. These learned weights guide the calibration of a JEPA world model's latent space, leading to significantly better image-based planning without requiring manual feature engineering or extra computation at test time.

What Problem Does This Paper Address?

Joint-Embedding Predictive Architectures (JEPAs) are a powerful class of world models that learn to predict future latent embeddings rather than reconstructing high-dimensional pixels. While efficient for planning, JEPAs often struggle because their visual prediction objectives don't guarantee that the latent space captures the specific physical properties needed for a task.

A common fix is to supervise the latent space using simulator states (e.g., object coordinates, joint angles). However, raw simulator states are often messy: they contain redundant information (like sine-cosine pairs for the same angle) and irrelevant "distractor" variables. Standard methods weight all these features equally, which distorts the latent geometry and leads to poor planning. Manually selecting which features "matter" is a tedious process that requires expert domain knowledge. SWIM aims to automate this selection process by looking at the agent's behavior.

Methods

SWIM operates in two distinct stages:

  1. Stage 1: Learning Behavioral Weights. The authors train a gated inverse-dynamics model. This model takes the current state $q_t$ and a "gated" future state displacement $g \odot (q_{t+h} - q_t)$ to predict the action $a_t$ that was taken. The gates $g$ are learnable parameters. If a specific state feature's change is highly informative for predicting the action, its corresponding gate value increases. To prevent the model from ignoring the displacement, they use a discrimination loss (like InfoNCE) that forces the model to distinguish between matching and mismatched future states.
  2. Stage 2: Calibrating Latent Geometry. The learned gates $g^*$ are frozen and used to define a weighted state distance metric. This metric is then used to train the JEPA's image encoder via the SCALE objective. SCALE encourages the Euclidean distance between latent embeddings to correlate strongly with the behaviorally-weighted state distances.

Crucially, the inverse-dynamics model and simulator states are only used during training. At deployment, the model plans directly from images using standard search algorithms like the Cross-Entropy Method (CEM).

Figure 1: SWIM. (a) The behavioral predictor learns coordinate weights from current states, gated future changes, and logged actions. (b) Frozen weights rescale standardized states. Squared latent and state distances from the same frame pairs enter the correlation loss, with state distances held fixed; prediction and SIGReg losses are retained. The inset names physical features before standardization and shows illustrative weight bars; within this inset, θ denotes a joint angle. The training symbols are defined in Sections 4.1–4.2. Deployment uses image-based latent planning; the behavioral predictor and logged future states are training-only components.

Datasets

The authors evaluated SWIM on OGBench, a suite of simulated robotic manipulation tasks. They focused on five specific environments:

  • Cube, Cube-Double, Cube-Triple, Cube-Quadruple: Tasks involving manipulating one to four blocks. These are ideal for testing weighting because the state dimension increases (from 22 to 37) as more objects are added.
  • Scene: A more complex environment (26-d state) involving articulated elements like drawers and buttons.

Key Insights and Ideas

  • Actions as a Proxy for Relevance: The core "aha moment" is that an agent's actions are the ultimate ground truth for what matters. If a state change doesn't help explain the action taken, it's likely redundant or irrelevant to the task.
  • Geometry is Planning: The paper reinforces the idea that for JEPAs, the geometry of the latent space is the planning cost. By aligning this geometry with a behaviorally-informed metric, the planning cost becomes a much more accurate signal of task progress.
  • Automatic Distractor Rejection: SWIM acts as an automated feature selector. In experiments where irrelevant "distractor" variables were intentionally added to the state, SWIM successfully assigned them near-zero weights, preserving planning performance where uniform weighting failed.

Key Results

  • Consistent Improvement: SWIM outperformed uniform state weighting (SCALE-All) across all five OGBench tasks, improving planning success by 1.0–6.0 percentage points.
  • Baseline Gains: Compared to a standard LeWorldModel (LeWM) without state supervision, SWIM improved success rates by up to 8.9 points.
  • Robustness: In "contaminated" scenarios (adding 26 unrelated variables to a 22-d state), SWIM maintained a success rate of ~75%, while uniform weighting dropped significantly.
  • Interpretability: The learned weights were found to be highly intuitive. For example, in block manipulation, the model prioritized end-effector height and block positions while downweighting redundant trigonometric joint encodings.

Limitations and Future Work

  • Behavioral Bias: The learned weights are only as good as the behaviors present in the offline dataset. If the data doesn't cover certain task-relevant movements, the model might miss important features.
  • Static Weighting: Currently, SWIM learns a single global weight vector for an entire task. The authors note that state-dependent weighting—where different features matter at different stages of a task—is a promising but more complex future direction.
  • Integration with Other Objectives: While demonstrated with SCALE, the authors suggest that behaviorally-learned weights could improve other state-grounding techniques, such as subspace alignment or physical-state grounding.

Summary updated

Abstract

Joint-embedding predictive architectures (JEPAs) enable world models to plan from images by predicting future embeddings. However, their visual prediction objectives do not explicitly incorporate task-relevant state information, so the physical state that determines task progress may not be reliably captured in the latent space. Supervising world models with simulator state can make planning more informative, but it requires deciding which state features matter. Features are typically selected by hand or weighted uniformly, and a uniform weighting gives every standardized state feature equal influence, so redundant coordinates can overcount the same variation, while irrelevant ones can obscure task-relevant differences. To address this limitation, we propose SWIM (State Weighting with Inverse-Dynamics Model), which learns state weights from offline trajectories by predicting recorded actions from current states and observed future displacements. The learned weights quantify each feature's contribution to action prediction and define a weighted state metric for state-supervised training, which we demonstrate by aligning latent and state distances. The inverse-dynamics model is used only during training, so the resulting world model plans from images with no additional planning-time computation. Across five simulated robotic manipulation tasks, SWIM consistently improves planning success over uniform state weighting.
BibTeX
@misc{hu2026swim,
  author = {Jiaming Hu and Shi Bo and Yan Zheng and Tian Wang and Florian Dubost and Alejandro Mottini and Junze Liu and Arvind Srinivasan and Kai Zhong and Kun Qian and Sharon Gao and Vianne Gao and Houyu Zhang and Qingjun Cui},
  title = {{SWIM: Learning What Matters from Behaviors for JEPA World Models}},
  year = {2026},
  eprint = {2610.36662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36662},
}
Figure