TL;DR
SWIM (State Weighting with Inverse-Dynamics Model) is a two-stage framework that automatically learns which physical state features are task-relevant by training an inverse-dynamics model on offline data. These learned weights guide the calibration of a JEPA world model's latent space, leading to significantly better image-based planning without requiring manual feature engineering or extra computation at test time.
What Problem Does This Paper Address?
Joint-Embedding Predictive Architectures (JEPAs) are a powerful class of world models that learn to predict future latent embeddings rather than reconstructing high-dimensional pixels. While efficient for planning, JEPAs often struggle because their visual prediction objectives don't guarantee that the latent space captures the specific physical properties needed for a task.
A common fix is to supervise the latent space using simulator states (e.g., object coordinates, joint angles). However, raw simulator states are often messy: they contain redundant information (like sine-cosine pairs for the same angle) and irrelevant "distractor" variables. Standard methods weight all these features equally, which distorts the latent geometry and leads to poor planning. Manually selecting which features "matter" is a tedious process that requires expert domain knowledge. SWIM aims to automate this selection process by looking at the agent's behavior.
Methods
SWIM operates in two distinct stages:
- Stage 1: Learning Behavioral Weights. The authors train a gated inverse-dynamics model. This model takes the current state $q_t$ and a "gated" future state displacement $g \odot (q_{t+h} - q_t)$ to predict the action $a_t$ that was taken. The gates $g$ are learnable parameters. If a specific state feature's change is highly informative for predicting the action, its corresponding gate value increases. To prevent the model from ignoring the displacement, they use a discrimination loss (like InfoNCE) that forces the model to distinguish between matching and mismatched future states.
- Stage 2: Calibrating Latent Geometry. The learned gates $g^*$ are frozen and used to define a weighted state distance metric. This metric is then used to train the JEPA's image encoder via the SCALE objective. SCALE encourages the Euclidean distance between latent embeddings to correlate strongly with the behaviorally-weighted state distances.
Crucially, the inverse-dynamics model and simulator states are only used during training. At deployment, the model plans directly from images using standard search algorithms like the Cross-Entropy Method (CEM).
Datasets
The authors evaluated SWIM on OGBench, a suite of simulated robotic manipulation tasks. They focused on five specific environments:
- Cube, Cube-Double, Cube-Triple, Cube-Quadruple: Tasks involving manipulating one to four blocks. These are ideal for testing weighting because the state dimension increases (from 22 to 37) as more objects are added.
- Scene: A more complex environment (26-d state) involving articulated elements like drawers and buttons.
Key Insights and Ideas
- Actions as a Proxy for Relevance: The core "aha moment" is that an agent's actions are the ultimate ground truth for what matters. If a state change doesn't help explain the action taken, it's likely redundant or irrelevant to the task.
- Geometry is Planning: The paper reinforces the idea that for JEPAs, the geometry of the latent space is the planning cost. By aligning this geometry with a behaviorally-informed metric, the planning cost becomes a much more accurate signal of task progress.
- Automatic Distractor Rejection: SWIM acts as an automated feature selector. In experiments where irrelevant "distractor" variables were intentionally added to the state, SWIM successfully assigned them near-zero weights, preserving planning performance where uniform weighting failed.
Key Results
- Consistent Improvement: SWIM outperformed uniform state weighting (SCALE-All) across all five OGBench tasks, improving planning success by 1.0–6.0 percentage points.
- Baseline Gains: Compared to a standard LeWorldModel (LeWM) without state supervision, SWIM improved success rates by up to 8.9 points.
- Robustness: In "contaminated" scenarios (adding 26 unrelated variables to a 22-d state), SWIM maintained a success rate of ~75%, while uniform weighting dropped significantly.
- Interpretability: The learned weights were found to be highly intuitive. For example, in block manipulation, the model prioritized end-effector height and block positions while downweighting redundant trigonometric joint encodings.
Limitations and Future Work
- Behavioral Bias: The learned weights are only as good as the behaviors present in the offline dataset. If the data doesn't cover certain task-relevant movements, the model might miss important features.
- Static Weighting: Currently, SWIM learns a single global weight vector for an entire task. The authors note that state-dependent weighting—where different features matter at different stages of a task—is a promising but more complex future direction.
- Integration with Other Objectives: While demonstrated with SCALE, the authors suggest that behaviorally-learned weights could improve other state-grounding techniques, such as subspace alignment or physical-state grounding.
Summary updated