WA-JEPA: Goal-Aware Predictive Feedback Action Refinement for a World-Action Model

Tao An, Weihang Liang, Tianqi Wang, Chuqing Cao, Wenming Tan, Kai Zhang, Haibo Lei, Tao Xie, Lijun Zhao, Ruifeng LiarXiv 2026 · arXiv

TL;DR

WA-JEPA is a novel world-action model that achieves high-performance robot control by iteratively refining action plans through "predictive feedback." By comparing latent future predictions to a task goal, it eliminates the need for expensive candidate-action search, resulting in a 10–32x speedup over traditional world-model planners while maintaining state-of-the-art success rates.

What Problem Does This Paper Address?

In robot learning, there is a persistent tension between planning and acting.

Traditional world models (like LeWM or DINO-WM) are great at "thinking ahead." They use a simulator to predict the consequences of actions and then use search algorithms like the Cross-Entropy Method (CEM) to sample thousands of possible action sequences to find the best one. While effective, this is computationally expensive and slow—often too slow for real-time reactive control.

On the other hand, imitation learning models (like ACT or Diffusion Policy) are fast. They map observations directly to actions. However, they don't explicitly reason about the consequences of their actions during inference. If the model makes a slight error in its initial proposal, it has no internal mechanism to "see" that the resulting future won't match the goal and correct itself before moving the robot.

The authors of WA-JEPA ask: Can we get the foresight of a planner with the speed of a direct action generator?

Methods

WA-JEPA (World-Action Joint-Embedding Predictive Architecture) bridges this gap using an iterative refinement loop. Instead of sampling thousands of random actions, it generates one proposal and "polishes" it.

  1. Latent Representation: The model uses a frozen DINOv2 encoder to turn images (observations and goals) into latent "scene tokens." This avoids the heavy cost of reconstructing pixels.
  2. The Proposal-Prediction-Feedback Loop:
    • Proposal: The model generates an initial sequence of actions.
  3. Prediction: A shared world model predicts the future latent scenes and proprioceptive states that would result from those actions.
  4. Feedback: The model compares these predicted futures to the actual goal image in the latent space.
  5. Refinement: Two specific feedback paths update the action plan:
  6. Goal-Discrepancy Residual Feedback (GDRF): It calculates the "gap" between the predicted outcome and the goal, using this error to apply chunk-wise corrections to the actions.
  7. Future-Scene Context Feedback (FSCF): It provides fine-grained spatial context from the predicted future to help the model coordinate complex movements.
  8. Joint Learning: The model is trained offline on demonstrations to simultaneously learn how the world works (dynamics) and how to fix its own plans (refinement).

Datasets

The authors evaluated WA-JEPA across several standard robotic benchmarks:

  • Short-Horizon: PushT (planar pushing), TwoRoom (navigation), and OGBench-Cube (manipulation).
  • Long-Horizon: The "Anchored Planning" framework, which requires reaching distant goals by retrieving intermediate visual waypoints.
  • Full-Task Execution: Push-T and LIBERO-Goal, a suite of 10 diverse manipulation tasks (e.g., opening drawers, picking up items).

Key Insights and Ideas

  • Search-less Planning: The biggest "aha" moment is that you don't need to evaluate 1,000 trajectories to find a good one. If your model understands the relationship between actions and outcomes, it can simply "nudge" a single trajectory in the right direction.
  • Explicit Discrepancy: The authors found that explicitly feeding the difference (Goal - Prediction) into the refinement module performed better than just concatenating the two. This makes the "error signal" clear to the model.
  • Efficiency via Latents: By performing all reasoning in the DINOv2 latent space, the model stays compact. WA-JEPA has significantly fewer trainable parameters (~14M-37M) compared to many Vision-Language-Action (VLA) models that reach into the billions.

Key Results

  • Massive Speedup: WA-JEPA is 10.8x to 32.6x faster than LeWM (a search-based world model). On the PushT task, it generates a plan in ~32ms, compared to over 1,000ms for the baseline.
  • Computational Thrift: It uses only 1.9% to 5.6% of the FLOPs (floating-point operations) required by search-based planners.
  • High Success: Despite the speed, it doesn't sacrifice performance. It achieved 96-100% success on short-horizon tasks and 97% success on the LIBERO-Goal benchmark, outperforming several much larger VLA and WAM baselines.
  • Refinement Works: Ablation studies showed that success on PushT jumped from 16% (with zero refinement) to 96% (with three refinement steps), proving that "thinking twice" is vital.

Limitations and Future Work

  • Visual Goal Dependency: The current model relies on a goal image. In many real-world scenarios, robots need to follow language instructions ("pick up the red mug"), which would require integrating a language backbone.
  • Offline Data Quality: Like all imitation learning methods, WA-JEPA is limited by the quality and diversity of the offline demonstrations it is trained on.
  • Fixed Refinement Steps: The model currently uses a fixed number of refinement iterations (usually 3). Future work could explore "anytime" algorithms where the model refines until a certain confidence threshold is met or until the next control cycle begins.

Summary updated

Abstract

JEPA-based world models predict action consequences in latent space, but representative planning approaches rely on candidate-action search, requiring repeated evaluation of action sequences and incurring substantial online computational cost. Direct action generation avoids this search, yet how to leverage latent predictions to guide action generation given task goals remains a key challenge in translating learned physical dynamics into effective decisions. We introduce WA-JEPA, a world-action model that couples action generation, action-conditioned latent prediction, and goal-aware feedback. The model first generates an action proposal and predicts the resulting future scenes and proprioceptive states, then refines the proposal through two complementary feedback paths, using the task goal as a reference. Goal-discrepancy residual feedback (GDRF) uses differences between the latent representations of predicted scenes and the task goal to guide chunk-wise action refinement. Future-scene context feedback (FSCF) preserves fine-grained predicted scene representations to guide subsequent action refinement. Jointly learned from offline demonstrations, this process iteratively updates the same plan before execution, making future predictions directly participate in action generation without candidate-action search. We evaluate short-horizon goal reaching, long-horizon tasks, and full-task execution. The results show that goal-aware predictive feedback supports competitive task performance while maintaining a compact model and low online computational cost. Code is available at https://github.com/Dovisya/WA-JEPA.
BibTeX
@misc{an2026wa,
  author = {Tao An and Weihang Liang and Tianqi Wang and Chuqing Cao and Wenming Tan and Kai Zhang and Haibo Lei and Tao Xie and Lijun Zhao and Ruifeng Li},
  title = {{WA-JEPA: Goal-Aware Predictive Feedback Action Refinement for a World-Action Model}},
  year = {2026},
  eprint = {2611.19183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19183},
}
Figure