Stuck in Place: Stall Diagnosis and Remedies for JEPA Planners

Junhyeok Lee, Dongheon Lee, Kyu Sung ChoiarXiv 2026 · arXiv

TL;DR

Researchers have discovered that failures in state-of-the-art JEPA-based robot planners are often caused by the planning loop getting stuck in a "fixed point" stall rather than just model error. They propose Unstick, a set of lightweight remedies—including episodic residual memory and random re-initialization—that recover most of the performance gains of expensive test-time adaptation without needing to update model weights during deployment.

What Problem Does This Paper Address?

Joint-Embedding Predictive Architectures (JEPAs) have become a popular way for robots to plan complex tasks directly from pixels. By predicting future latent embeddings rather than raw pixels, these models are efficient and robust. However, they frequently fail even on tasks they should theoretically handle.

The standard diagnosis for these failures is "distribution shift"—the idea that the model encounters visual or physical conditions it didn't see during training. The standard cure is test-time adaptation, where the model's weights are updated online as the robot acts.

This paper argues we’ve been looking at the wrong culprit. By analyzing the closed-loop behavior of planners on pushing tasks, the authors found that at least 96% of failures are "stalls." In a stall, the robot predicts it is making progress toward a goal, but in reality, it is just idling or moving back and forth without moving the target object. This happens because the planning loop—specifically when using warm-started Gradient Descent (GD)—reaches a mathematical fixed point where it keeps proposing the same ineffective plan over and over.

Figure 1. Latent costs and frames of one PushObj-T episode. Top: base GD planner; bottom: Unstick-Mem (weights fixed at test time). Solid: actual latent cost; dashed: predicted after each chunk; arrow: predicted drop per chunk; gray band: stall. Frames: last step; blue: agent; red: agent path during the stall; dashed outline: goal.

Methods

The authors introduce Unstick, a framework that pairs on-policy fine-tuning (FT) before deployment with two "loop-breaking" strategies that keep model weights fixed at test time:

  1. On-policy Fine-tuning (FT): Before the robot is deployed, it is allowed to run its base planner on a few episodes. The model is then fine-tuned on the resulting transitions. This helps the model "see" the specific states and failures its own planner tends to produce.
  2. Unstick-Mem (Residual Memory): This method maintains a cache of the model's one-step prediction errors (residuals) during the current episode. When planning, the model corrects its future predictions by adding back these stored errors if the proposed action is similar to one it already tried. This effectively "tells" the planner: "You thought this action would move the block, but it didn't; don't fall for it again."
  3. Unstick-ReInit (Random Re-initialization): Standard planners use a "warm start," beginning the optimization for the next step with the tail end of the previous plan. While fast, this reinforces the fixed-point trap. Unstick-ReInit simply starts every replanning step from a fresh random plan, breaking the deterministic cycle.

Datasets

The methods were evaluated on three primary benchmarks:

  • PushObj: Pushing blocks of various shapes (T, L, Z, cross). Evaluation included "seen" shapes and "unseen" shapes (I-shape, square).
  • PushT: A standard benchmark for pushing a T-shaped block, tested under various visual shifts like blur, salt-and-pepper noise, darkening, and color changes.
  • PointMaze: A navigation task used to test dynamics shifts (changes in mass/damping) and layout shifts (unseen mazes).

Key Insights and Ideas

  • The Planning Loop is the Problem: The paper shifts the blame from "bad models" to "bad loops." Even a slightly imperfect model can cause a total failure if the planning algorithm (like warm-started GD) gets trapped in a cycle of over-optimism.
  • Stalls are Universal: Stalls weren't just a result of distribution shift; they occurred even on training shapes and clean frames. This suggests that "stalling" is an inherent degeneracy in latent MPC planners.
  • Fixed Weights are Sufficient: You don't need to perform expensive backpropagation to update model weights at test time to fix these issues. Simply breaking the loop's determinism or using a simple memory buffer can achieve similar results.
Figure 2. Failure analysis of the base GD planner. (a) Idle: agent moves <2 px per MPC step. (b, c) The 101 stalls at MPC step 25 (T, clean); (c) labels give counts. Objective trap: no simple plan lowers the simulated cost of staying by ≥5%; misranked: a simple plan lowers the simulated cost, but the model prefers staying.

Key Results

  • Recovering Adaptation Gains: On pushing tasks, Unstick-Mem and Unstick-ReInit recovered 80% and 82% of the success rate gains provided by AdaJEPA (a state-of-the-art test-time adaptation method), respectively.
  • Outperforming Adaptation on Unseen Shapes: For unseen block shapes (like the square), Unstick-ReInit actually outperformed test-time adaptation.
  • Efficiency: Unstick methods are computationally lighter than test-time adaptation because they don't require a backward pass (gradient calculation) during the robot's operation. Unstick-Mem adds only a small overhead for memory lookup, while Unstick-ReInit adds no extra computation at all.
  • Visual Shifts: While Unstick-Mem handled visual shifts well (recovering 72% of adaptation gains), ReInit struggled more there, suggesting that visual shifts do require some form of model correction (which the memory provides).

Limitations and Future Work

  • Encoder Failures: The authors noted that neither remedy helps when the encoder itself discards vital information. For example, if a visual shift makes the block's orientation unreadable, the planner will still fail because the underlying "map" is broken.
  • Long Episodes: The current residual memory grows with the episode length. For very long-running tasks, a strategy to prune or bound the memory would be necessary.
  • Search Methods: The diagnosis focused heavily on Gradient Descent planners. While the authors briefly touched on Cross-Entropy Method (CEM) planners, they found that CEM fails differently (the agent moves, but the block still stalls), suggesting further diagnostic work is needed for sampling-based planners.

Summary updated

Abstract

Joint-embedding predictive architecture (JEPA) world models plan from pixels with model predictive control, yet often fail on tasks their model should handle. Test-time adaptation attributes such failures to model error under distribution shift and updates the weights on the current episode, without checking this attribution against the planning loop itself. We follow the executed planning loop on pushing tasks. In most settings, with and without distribution shift, nearly all failures (at least 96%) are stalls, where the object stays in place while the model predicts progress. The warm-started gradient-descent (GD) planner then returns nearly the same plan, a fixed point where the agent mostly idles. We propose Unstick, which pairs on-policy fine-tuning before deployment with a fixed-weight loop breaker: a residual memory of the episode's prediction errors or random re-initialization at every replan. With GD, Unstick recovers most of the adaptation gain on pushing tasks with the memory, and with re-initialization most of it on shape shifts but much less on visual shifts. A large part of what limits these planners under GD is thus a fixed point of the planning loop, present with and without distribution shift, which a planner with fixed weights can break. Code is available at https://github.com/jhlee0619/unstick.
BibTeX
@misc{lee2026stuck,
  author = {Junhyeok Lee and Dongheon Lee and Kyu Sung Choi},
  title = {{Stuck in Place: Stall Diagnosis and Remedies for JEPA Planners}},
  year = {2026},
  eprint = {2611.25415},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25415},
}
Figure