Manufacturing Differential Sparsity: Planner-Hardware Co-Design for Efficient World Model Planning

Wenbo Zhu, Paul Yi-Chia Chen, Yuanhan Li, Zishen Wan, Christopher TorngarXiv 2026 · arXiv

This paper introduces Candelta, a hardware-algorithm co-design that significantly accelerates planning in Joint Embedding Predictive Architecture (JEPA) world models. By observing that similar actions lead to similar predicted states ("latent locality"), the authors developed a planner that "manufactures" bit-level sparsity and a custom accelerator that exploits this sparsity to achieve up to 1.80× speedup and 52% energy savings over dense hardware.

TL;DR

Candelta speeds up embodied AI planning by forcing candidate trajectories to be mathematically similar, creating "differential bit-level sparsity" that custom hardware can skip. This co-design makes JEPA-style world model planning—previously too slow for real-time use—significantly more efficient without sacrificing robot performance.

What Problem Does This Paper Address?

For a robot to navigate a room or move an object, it needs to plan. JEPA-style world models are a powerful way to do this: they predict how the world’s "latent" (hidden) state evolves based on actions. However, planning usually involves Model Predictive Control (MPC), which is computationally brutal.

In MPC, a planner might sample hundreds of possible action sequences and "roll them out" through the world model to see which one reaches the goal. This process repeats for many iterations. For a single planning step, a model might require 45,000 predictor calls, taking tens of seconds even on a high-end GPU. This latency makes real-time embodied control nearly impossible.

Methods

The authors propose a co-design consisting of two parts:

  1. The Candelta Planner: Standard planners (like the Cross-Entropy Method, or CEM) sample trajectories independently. Candelta instead uses Reference Forest Sampling. It organizes candidates into a "forest" where "child" trajectories are mathematically coupled to "root" trajectories. This ensures that many candidates are very similar to one another.
  2. The Candelta Accelerator: This is a custom bit-sparse hardware design. Instead of processing full numbers, it computes the difference (delta) between a candidate and its "parent" in the reference forest.

Because the planner forces these candidates to be similar, the "delta" between them contains a huge number of zero bits. The accelerator uses a "term-serial" design, meaning it only spends clock cycles on the non-zero bits. If a delta has many zeros, the hardware finishes the calculation much faster.

Figure 1. Planner and accelerator design for efficient planning. (a) A sampling-based planning event: ①Planner samples P action sequences; ②P candidates roll out through H sequential predictor calls; ③Costs are evaluated; ④The planner refits N(µ, σ2) from the E elites. ①-④are repeated for T iterations. (b) Standard CEM rolls out trajectories by sampling the action sequences independently. (c) The Candelta planner draws from the same sampling distribution as a reference forest: the action sequence of each child is coupled to that of its group’s root, while every sampled action sequence keeps the CEM marginal, creating hierarchically correlated candidate trajectories. (d) Each child activation is encoded against its root’s, and their roots against the mean candidate.

Datasets

The authors evaluated their design across three different JEPA-style world models:

  • LeWM (Latent World Model): A compact model (18M parameters) that encodes observations into a single token.
  • DINO-WM: Uses DINOv2 patch features.
  • JEPA-WM: A larger model used for robotic manipulation.

Evaluation tasks included standard benchmarks like Push-T (pushing a T-shaped block), OGBench Cube, TwoRoom, and Reacher.

Key Insights and Ideas

  • Latent Locality: The core "aha!" moment is that the predictor's latent space is smooth. Small changes in action input result in small, predictable changes in the latent output. Figure 3. Visualizing latent locality: a grid of actions (left) maps to a structured, "stretched" grid of predicted next-state latents (right).

  • Manufacturing Sparsity: Usually, engineers treat sparsity as a fixed property of a model or dataset. This paper argues that the algorithm (the planner) can choose to create more sparsity by how it samples, effectively "shaping" the workload for the hardware.

  • The Mean Anchor: The authors found that the "mean" candidate of a sampling distribution is a near-optimal reference. Encoding other candidates against this mean significantly reduces the number of bits the hardware needs to process.
  • Adaptive Coupling: To prevent the planner from getting "stuck" (losing exploration), the coupling between children and roots is adaptive. It starts weak to allow broad searching and gets stronger as the planner converges on a solution.
Figure 3. Left: actions around the sampling mean (×) on a plane through two random directions drawn from the CEM sampling distribution. Right (1–6): 2-D PCA projection of the next-state latents that the predictor computes for these actions from the same current state, for six real LeWM CEM iterations on OGBench Cube (Park et al., 2025), each on its own scale.

Key Results

  • Speed & Efficiency: Candelta is 1.28–1.80× faster than a dense INT8 accelerator of the same area and uses 38–52% less energy.
  • Beating Prior Art: It outperforms "Pragmatic" (a state-of-the-art bit-sparse design) by 1.22–1.58× in speed while using 16–36% less energy.
  • Real-world Comparison: Compared to an NVIDIA Jetson Thor (a high-end commercial SoC for robotics), Candelta uses 7.2× to 17.2× less energy per planning event.
  • No Quality Loss: Crucially, the "manufactured" similarity does not hurt the robot's success rate on tasks like Push-T.
Figure 4. (a) Impact to planning quality after applying quantization with the 95% CI. (b) Differential bit-level sparsity when encoded against a reference. Measured on Push-T (Chi et al., 2023).

Limitations and Future Work

  • The Weight Stream Floor: The speedup is ultimately limited by the time it takes to stream weights from memory. Even if an activation has zero bits, the hardware still needs to fetch the corresponding weights. The authors note a theoretical speedup ceiling of ~2.46× for this specific architecture.
  • SRAM Overhead: Storing the "parent" activations to compute deltas requires on-chip memory (SRAM), which could be a constraint for very large models.
  • Algorithm Scope: While demonstrated on CEM, the authors suggest this approach could extend to other population-based planners like MPPI (Model Predictive Path Integral control), which are also common in robotics.

Summary updated

Abstract

JEPA-style latent world models let embodied agents plan toward new goals with model predictive control, but they are still too slow for real-life embodied control: sampling-based MPC repeats predictor inference tens of thousands of times per plan, and even on high-end GPUs one planning step can take tens of seconds. We observe substantial latent locality in this repeated-inference workload: the predictor maps close actions to close latents, and inside a quantized predictor this closeness takes a form that hardware can exploit, differential bit-level sparsity. Unlike an input stream, whose temporal similarity is given, a planner itself decides which candidates it evaluates together, so the planner can choose its own references and, with reference forest sampling, manufacture more sparsity: it couples the sampled action sequences of each group to a shared root, with competitive planning quality. We build the accelerator on the insights of the rich line of bit-level sparsity accelerators and design it and the planner tightly coupled as a single co-design: the planner increases sparsity and exposes structure for hardware scheduling, and the bit-exact hardware effectively uses the sparsity that the planner creates. On three JEPA world models, our co-design is 1.28-1.80x faster than a dense accelerator and uses 38-52% less energy, and it is 1.22-1.58x faster than prior bit-sparse design and uses 16-36% less energy, with the same area. To our knowledge, our planner is the first planner that manufactures differential bit-level sparsity across its candidates and exposes its structure for hardware scheduling, and our co-design is the first algorithm-hardware co-design for JEPA-style world-model planning.
BibTeX
@misc{zhu2026manufacturing,
  author = {Wenbo Zhu and Paul Yi-Chia Chen and Yuanhan Li and Zishen Wan and Christopher Torng},
  title = {{Manufacturing Differential Sparsity: Planner-Hardware Co-Design for Efficient World Model Planning}},
  year = {2026},
  eprint = {2611.22483},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22483},
}
Figure