Spatial Mixture-of-Experts

Nikoli Dryden, Torsten Hoefler

Abstract

Many data have an underlying dependence on spatial location; it may be weather on the Earth, a simulation on a mesh, or a registered image. Yet this feature is rarely taken advantage of, and violates common assumptions made by many neural network layers, such as translation equivariance. Further, many works that do incorporate locality fail to capture fine-grained structure. To address this, we introduce the Spatial Mixture-of-Experts (SMoE) layer, a sparsely-gated layer that learns spatial structure in the input domain and routes experts at a fine-grained level to utilize it. We also develop new techniques to train SMoEs, including a self-supervised routing loss and damping expert errors. Finally, we show strong results for SMoEs on numerous tasks, and set new state-of-the-art results for medium-range weather prediction and post-processing ensemble weather forecasts.

Introduction

In the field of computer vision and spatial modeling, the principle of translation equivariance has long been a cornerstone of architectural design. Convolutional Neural Networks (CNNs) apply the same filters across an entire image, assuming that a feature’s importance is independent of its coordinates. However, many real-world datasets—ranging from global climate models to medical imaging—fundamentally violate this assumption. In weather forecasting, for instance, the atmospheric dynamics at the equator differ significantly from those at the poles.

While Locally-Connected Networks (LCNs) can model these variations by using unique parameters for every location, they suffer from a massive explosion in parameter counts, making them computationally prohibitive for high-resolution tasks. To bridge this gap, we introduce the Spatial Mixture-of-Experts (SMoE) layer. SMoE provides the flexibility of local modeling with the efficiency of sparse computation, allowing networks to develop a "sense of place."

The Spatial Mixture-of-Experts (SMoE) Layer

The core innovation of SMoE is its ability to learn and exploit location-dependent patterns through fine-grained routing. Unlike traditional Mixture-of-Experts (MoE) models that route entire images or large patches to specific experts, SMoE operates at the per-pixel level.

Fine-Grained Tensor Routing

In an SMoE layer, each spatial location $(h, w)$ independently selects the most appropriate expert from a shared pool. To minimize the computational overhead of this decision-making process, we utilize Tensor Routing. Instead of a complex neural network gate, we use a learnable spatial tensor $D$ that directly encodes routing preferences. During the forward pass, the gate simply performs a lookup, requiring zero additional FLOPs to decide which expert to invoke for a given coordinate.

Stabilizing Training for Regression

Training sparse gates for regression tasks (such as temperature prediction) is notoriously difficult because the error signal does not explicitly indicate which expert should have been chosen. We address this with two novel techniques:

  1. Routing Classification (RC) Loss: A self-supervised loss that treats routing as a classification task. It uses the magnitude of the error signal as a proxy for "incorrectness," guiding the gate to prefer experts that minimize local error.
  2. Expert Error Damping: To prevent experts from being corrupted by gradients from misrouted pixels, we damp the error signal for incorrect routings, ensuring that experts only learn from the data they are best suited to handle.

Experimental Evaluation

Synthetic Benchmarks: Heat Diffusion

To test if SMoE can recover hidden spatial structures, we designed a Location-Dependent Heat Diffusion task where the rate of heat spread varies by region. While standard CNNs converged to an "average" diffusivity—failing to capture the underlying spatial map—SMoE successfully recovered the distinct regions, matching the performance of parameter-heavy LCNs while remaining computationally efficient.

Weather Forecasting and Post-Processing

We evaluated SMoE on WeatherBench, the industry standard for medium-range global weather prediction. SMoE achieved new state-of-the-art results for geopotential (Z500) and temperature (T850) metrics. Notably, it outperformed models pretrained on over a century of climate simulation data, demonstrating that architectural awareness of spatial locality can be more powerful than massive data scaling alone.

On the ENS-10 dataset, which involves post-processing ensemble forecasts to correct systematic biases, SMoE significantly outperformed high-resolution U-Net baselines. This confirms the model's ability to scale to complex, high-dimensional spatial domains.

Generalization to Image Classification

To ensure SMoE does not degrade performance on data with less rigid spatial structures, we tested it on ImageNet-1k. SMoE matched or exceeded the performance of Low-Rank Locally Connected Networks (LRLCN) while utilizing 44% fewer parameters, proving that sparsity is an effective tool for relaxing translation equivariance without sacrificing efficiency.

Conclusion and Future Work

Spatial Mixture-of-Experts represents a significant step toward neural networks that can adapt to the inherent geometry of their data. By moving beyond the "one-size-fits-all" approach of standard convolutions, SMoE allows for specialized processing that respects the unique physics and structures of different spatial regions.

While SMoE is theoretically efficient in terms of FLOPs, current deep learning frameworks are optimized for dense operations. Future work will focus on developing custom CUDA kernels to eliminate the overhead of "gather/scatter" operations. Additionally, we aim to extend SMoE to general graphs, potentially revolutionizing how we process 3D meshes and social network data.

Summary updated

Abstract

Many data have an underlying dependence on spatial location; it may be weather on the Earth, a simulation on a mesh, or a registered image. Yet this feature is rarely taken advantage of, and violates common assumptions made by many neural network layers, such as translation equivariance. Further, many works that do incorporate locality fail to capture fine-grained structure. To address this, we introduce the Spatial Mixture-of-Experts (SMoE) layer, a sparsely-gated layer that learns spatial structure in the input domain and routes experts at a fine-grained level to utilize it. We also develop new techniques to train SMoEs, including a self-supervised routing loss and damping expert errors. Finally, we show strong results for SMoEs on numerous tasks, and set new state-of-the-art results for medium-range weather prediction and post-processing ensemble weather forecasts.
BibTeX
@inproceedings{dryden2022spatial,
  author = {Nikoli Dryden and Torsten Hoefler},
  title = {{Spatial Mixture-of-Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=eOVcWj4AXX},
}
Figure