SpikWM: Efficient End-to-End Spiking World Modeling for Visual Planning

Shuhan Ye, Hongbin Yu, Chong Wang, Hui Lu, Jiaming He, Yi Yu, Jiangbo Qian, Xudong JiangarXiv 2026 · arXiv

TL;DR

SpikWM is the first end-to-end spiking latent world model designed for efficient goal-conditioned visual planning. By introducing novel spiking architectures for encoding and prediction, it achieves success rates comparable to dense artificial neural networks (ANNs) while reducing estimated arithmetic energy consumption by over 75%.

What Problem Does This Paper Address?

Visual planning—the ability of a robot to "imagine" a sequence of actions to reach a goal—is computationally expensive. Traditional World Models (WMs) use dense ANNs to predict future states, but during planning, these models must be rolled out thousands of times to evaluate candidate plans. This leads to high latency and energy consumption, which is problematic for edge devices or robots with limited power.

Spiking Neural Networks (SNNs) offer a bio-inspired, energy-efficient alternative because they communicate via sparse, binary "spikes." However, applying SNNs to visual planning is difficult. A single spike can be too "blunt" to capture subtle motion cues, and the binary nature of SNNs often leads to "action collisions," where different actions result in the same silent neural response, making it impossible for a planner to distinguish between them.

Figure 1 illustrates how standard SNNs lose motion cues and fail to distinguish between different actions compared to ANNs.

Methods

The authors propose SpikWM, a Spiking Latent World Model based on the Joint-Embedding Predictive Architecture (JEPA). Unlike models that reconstruct pixels, SpikWM predicts future states in a compact "latent" space. To overcome the inherent limitations of SNNs, they introduced three key modules:

  1. Spiking Regional Encoding (SRE): Standard global pooling in SNNs often discards spatial information. SRE uses regional pooling to maintain spatial cues, ensuring the model knows where objects are in the frame.
  2. Bi-Spike Action Encoding (BAE): To prevent action collisions, BAE uses two complementary spike streams—one for positive and one for negative binary responses. This ensures that even subtle differences in actions are preserved in the spiking signal.
  3. Causal Membrane Prediction (CMP): This is the "brain" of the world model. It uses membrane dynamics to carry "subthreshold" evidence (information that isn't strong enough to trigger a spike yet) across the latent history. It employs learned group-wise time constants and causal spike-driven attention to predict the next state.
The SpikWM architecture, highlighting the SRE, BAE, and CMP modules.

The model is trained end-to-end on reward-free image-action trajectories, learning to predict the "next embedding" rather than the next image.

Datasets

The model was evaluated on four diverse offline datasets:

  • Push-T: A robotic manipulation task requiring precise object pushing.
  • Two-Room: A navigation task involving moving between rooms.
  • Reacher: A joint-configuration control task for a robotic arm.
  • OGBench-Cube: A challenging cube placement task.

Key Insights and Ideas

  • Subthreshold Information Matters: One of the biggest "aha" moments is that a lack of a spike doesn't mean a lack of information. By carrying the membrane potential forward (CMP), SpikWM retains "evidence" of past states that can influence future spikes, effectively providing the model with a form of short-term memory.
  • Solving the Collision Problem: The Bi-Spike encoding (BAE) is a clever fix for a fundamental SNN weakness. By allowing for "negative" spikes, the model can distinguish between "no action" and "an action that results in a negative current," which would both be zero in a standard SNN.
  • Efficiency without Sacrifice: The paper demonstrates that you don't need high-frequency spiking (multiple simulation steps per image) to achieve high performance. SpikWM uses just one simulation step per image, maximizing efficiency.

Key Results

SpikWM significantly outperforms previous spiking baselines and remains competitive with state-of-the-art ANNs:

  • Success Rates: It achieved mean success rates of 88.67% (Push-T), 97.33% (Two-Room), 99.33% (Reacher), and 86.67% (OGBench-Cube).
  • Energy Efficiency: Under a matched planning budget, SpikWM reduced estimated arithmetic energy by 75.6% relative to the ANN baseline, LeWM.
  • Robustness: Controlled replays showed that SpikWM is more sensitive to physical state changes (like an object moving) than to simple appearance changes (like a color shift), which is a desirable trait for a world model.
Figure 3. Open-loop forecasts on four tasks. Frame pairs show steps 15/20 (Push-T), 30/35 (Two-Room), 20/35 (OGBench-Cube), and 25/30 (Reacher). We render state readouts of the latent forecasts under shared actions [6].

Limitations and Future Work

  • Hardware Realization: While the estimated arithmetic energy is much lower, the actual energy savings depend on running the model on specialized neuromorphic hardware, which was not the focus of this paper.
  • Offline Training: The model was trained on offline datasets. Extending this to online, real-time learning in a dynamic environment remains a future challenge.
  • Long-Horizon Planning: While effective for the tested tasks, the limits of the causal membrane prediction over very long temporal horizons (hundreds of steps) are not yet fully explored.

Summary updated

Abstract

Goal-conditioned planning with image observations and goals requires predictions that distinguish visual states and action consequences. Spiking neural networks offer sparse computation, but single-step spikes can obscure motion cues and merge actions with different physical outcomes. We propose the Spiking Latent World Model (SpikWM), an end-to-end joint-embedding predictive architecture (JEPA) with one simulation step per image and one neural update per latent position. Spiking Regional Encoding retains spatial cues through regional pooling. Bi-Spike Action Encoding combines positive and negative binary responses, while Causal Membrane Prediction carries subthreshold evidence across the latent history with learned group-wise time constants. The encoder and predictor are trained jointly on reward-free image--action trajectories using next-embedding prediction and SIGReg. With a common planner, SpikWM achieves mean success rates of $88.67\%$, $97.33\%$, $99.33\%$, and $86.67\%$ on Push-T, Two-Room, Reacher, and OGBench-Cube, respectively, outperforming two spiking baselines. Push-T ablations support each module, and controlled replays examine responses to appearance and physical state changes. Under a matched planning budget, SpikWM reduces estimated arithmetic energy by $75.6\%$ relative to LeWM. Code will be released at https://github.com/ShawnYE618/SpikWM.
BibTeX
@misc{ye2026spikwm,
  author = {Shuhan Ye and Hongbin Yu and Chong Wang and Hui Lu and Jiaming He and Yi Yu and Jiangbo Qian and Xudong Jiang},
  title = {{SpikWM: Efficient End-to-End Spiking World Modeling for Visual Planning}},
  year = {2026},
  eprint = {2611.22981},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22981},
}
Figure