TL;DR
SpikWM is the first end-to-end spiking latent world model designed for efficient goal-conditioned visual planning. By introducing novel spiking architectures for encoding and prediction, it achieves success rates comparable to dense artificial neural networks (ANNs) while reducing estimated arithmetic energy consumption by over 75%.
What Problem Does This Paper Address?
Visual planning—the ability of a robot to "imagine" a sequence of actions to reach a goal—is computationally expensive. Traditional World Models (WMs) use dense ANNs to predict future states, but during planning, these models must be rolled out thousands of times to evaluate candidate plans. This leads to high latency and energy consumption, which is problematic for edge devices or robots with limited power.
Spiking Neural Networks (SNNs) offer a bio-inspired, energy-efficient alternative because they communicate via sparse, binary "spikes." However, applying SNNs to visual planning is difficult. A single spike can be too "blunt" to capture subtle motion cues, and the binary nature of SNNs often leads to "action collisions," where different actions result in the same silent neural response, making it impossible for a planner to distinguish between them.
Methods
The authors propose SpikWM, a Spiking Latent World Model based on the Joint-Embedding Predictive Architecture (JEPA). Unlike models that reconstruct pixels, SpikWM predicts future states in a compact "latent" space. To overcome the inherent limitations of SNNs, they introduced three key modules:
- Spiking Regional Encoding (SRE): Standard global pooling in SNNs often discards spatial information. SRE uses regional pooling to maintain spatial cues, ensuring the model knows where objects are in the frame.
- Bi-Spike Action Encoding (BAE): To prevent action collisions, BAE uses two complementary spike streams—one for positive and one for negative binary responses. This ensures that even subtle differences in actions are preserved in the spiking signal.
- Causal Membrane Prediction (CMP): This is the "brain" of the world model. It uses membrane dynamics to carry "subthreshold" evidence (information that isn't strong enough to trigger a spike yet) across the latent history. It employs learned group-wise time constants and causal spike-driven attention to predict the next state.
The model is trained end-to-end on reward-free image-action trajectories, learning to predict the "next embedding" rather than the next image.
Datasets
The model was evaluated on four diverse offline datasets:
- Push-T: A robotic manipulation task requiring precise object pushing.
- Two-Room: A navigation task involving moving between rooms.
- Reacher: A joint-configuration control task for a robotic arm.
- OGBench-Cube: A challenging cube placement task.
Key Insights and Ideas
- Subthreshold Information Matters: One of the biggest "aha" moments is that a lack of a spike doesn't mean a lack of information. By carrying the membrane potential forward (CMP), SpikWM retains "evidence" of past states that can influence future spikes, effectively providing the model with a form of short-term memory.
- Solving the Collision Problem: The Bi-Spike encoding (BAE) is a clever fix for a fundamental SNN weakness. By allowing for "negative" spikes, the model can distinguish between "no action" and "an action that results in a negative current," which would both be zero in a standard SNN.
- Efficiency without Sacrifice: The paper demonstrates that you don't need high-frequency spiking (multiple simulation steps per image) to achieve high performance. SpikWM uses just one simulation step per image, maximizing efficiency.
Key Results
SpikWM significantly outperforms previous spiking baselines and remains competitive with state-of-the-art ANNs:
- Success Rates: It achieved mean success rates of 88.67% (Push-T), 97.33% (Two-Room), 99.33% (Reacher), and 86.67% (OGBench-Cube).
- Energy Efficiency: Under a matched planning budget, SpikWM reduced estimated arithmetic energy by 75.6% relative to the ANN baseline, LeWM.
- Robustness: Controlled replays showed that SpikWM is more sensitive to physical state changes (like an object moving) than to simple appearance changes (like a color shift), which is a desirable trait for a world model.
Limitations and Future Work
- Hardware Realization: While the estimated arithmetic energy is much lower, the actual energy savings depend on running the model on specialized neuromorphic hardware, which was not the focus of this paper.
- Offline Training: The model was trained on offline datasets. Extending this to online, real-time learning in a dynamic environment remains a future challenge.
- Long-Horizon Planning: While effective for the tested tasks, the limits of the causal membrane prediction over very long temporal horizons (hundreds of steps) are not yet fully explored.
Summary updated