Spatial Mixture-of-Experts

Nikoli Dryden, Torsten Hoefler

Many real-world datasets have an underlying dependence on spatial location. In weather forecasting, a physical process at the equator behaves differently than one at the poles; in facial recognition, eyes are consistently found in the same relative area.

Standard deep learning approaches, particularly Convolutional Neural Networks (CNNs), are built on the principle of translation equivariance. They assume that a feature should be processed the exact same way regardless of where it appears in the input. While "Locally-Connected Networks" (LCNs) exist to solve this by learning unique filters for every location, they suffer from a massive explosion in parameters.

To find a middle ground, researchers have introduced the Spatial Mixture-of-Experts (SMoE) layer: a sparsely-gated neural network component that learns spatial structure in the input domain and routes specialized "experts" at a fine-grained level.

SMoE balances inductive bias and parameter count, offering fine-grained routing compared to standard convolutions or locally-connected layers.

The SMoE Architecture

The core innovation of the SMoE layer is adapting the traditional "Mixture-of-Experts" (MoE) concept specifically for spatial dimensions.

  • Fine-Grained Routing: Unlike traditional MoEs that route entire images or large patches to experts, SMoEs route at the per-pixel level. Each location in the input grid selects a small subset of experts from a shared pool, allowing the model to learn highly localized "physics" or features.
  • Tensor Routing: Standard MoE gating networks can be computationally expensive. For data on fixed grids (like latitude/longitude maps), the gate doesn't necessarily need to "see" the input data to know which expert to use; it only needs the coordinates. SMoEs utilize a simple learnable tensor as a highly efficient "lookup table" for locations.

Training Innovations: The Gradient as a Teacher

Training discrete routing gates in MoEs is notoriously difficult, especially for regression tasks. To make SMoEs work, the authors introduced two novel training techniques:

  1. Routing Classification (RC) Loss: The authors developed a self-supervised loss that leverages the backpropagated error signal (gradient) of the model. If an expert produces a high error at a certain pixel, the RC loss teaches the gate that it made a "mistake" and should route to a different expert in the future. This elegantly turns a hard discrete choice into a searchable optimization problem.
  2. Expert Error Damping: To prevent experts from being "corrupted" by learning from locations they were accidentally or incorrectly routed to during early training, the gradients for incorrect routings are actively damped.
Overview of the SMoE training process, highlighting the Routing Classification Loss and Expert Error Damping mechanisms.

Key Results and Performance

The authors evaluated SMoEs across a variety of challenging spatial tasks, demonstrating significant improvements over traditional architectures:

  • Weather Forecasting SOTA: SMoEs achieved new state-of-the-art results on WeatherBench for medium-range (3-5 day) weather prediction, significantly outperforming standard ResNets and even models pretrained on 150 years of climate simulation data. They also excelled on the ENS-10 dataset for post-processing ensemble forecasts.
  • Synthetic Perfection: On a custom "Location-Dependent Heat Diffusion" benchmark where the physics of heat spread changes by region, SMoEs achieved 100% accuracy (within a 1% error margin). Standard CNNs and Transformers hovered around 91-93% because their rigid equivariance prevented them from adapting to changing regional diffusion rates.
  • Parameter Efficiency: On ImageNet-1k, SMoEs matched or beat Low-Rank Locally Connected Networks (LRLCN) while using significantly fewer parameters, proving their value even when spatial location is less strictly fixed.
  • Emergent Specialization: Visualizations of the routing gates showed that the model naturally learned to assign different experts to land versus sea, and to different latitudinal zones, entirely without explicit supervision.

Limitations and Future Directions

While SMoEs are theoretically "flop-efficient" (requiring less raw computation per forward pass), they involve irregular memory access patterns. Because current deep learning frameworks like PyTorch are highly optimized for dense, regular matrix multiplications, the current SMoE implementation is slower in wall-clock time than a standard CNN. Realizing their full speed potential will likely require custom CUDA kernels or hardware-accelerated sparsity support.

Furthermore, the current SMoE design is tailored for fixed Cartesian grids. A highly promising future direction is generalizing this spatial routing logic to Graph Neural Networks (GNNs), which would allow SMoEs to be applied to the irregular meshes commonly used in complex engineering and fluid dynamics simulations.

Summary updated

Abstract

Many data have an underlying dependence on spatial location; it may be weather on the Earth, a simulation on a mesh, or a registered image. Yet this feature is rarely taken advantage of, and violates common assumptions made by many neural network layers, such as translation equivariance. Further, many works that do incorporate locality fail to capture fine-grained structure. To address this, we introduce the Spatial Mixture-of-Experts (SMoE) layer, a sparsely-gated layer that learns spatial structure in the input domain and routes experts at a fine-grained level to utilize it. We also develop new techniques to train SMoEs, including a self-supervised routing loss and damping expert errors. Finally, we show strong results for SMoEs on numerous tasks, and set new state-of-the-art results for medium-range weather prediction and post-processing ensemble weather forecasts.
BibTeX
@inproceedings{dryden2022spatial,
  author = {Nikoli Dryden and Torsten Hoefler},
  title = {{Spatial Mixture-of-Experts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=eOVcWj4AXX},
}
Figure