TL;DR
Most spatiotemporal prediction models rely on symmetric autoencoders, applying identical receptive field sizes to both encoding and decoding steps. This paper introduces ARFA (Asymmetric Receptive Field Autoencoder), which pairs a large-kernel encoder for global context extraction with a small-kernel decoder for local detail reconstruction. ARFA achieves state-of-the-art results across multiple benchmarks and introduces RainBench, a massive new radar echo dataset for precipitation forecasting.
What Problem Does This Paper Address?
Spatiotemporal prediction—predicting future frames in a sequence, such as video frames or weather patterns—requires deep neural networks to master both spatial structures and temporal transitions. The dominant architecture for this task is the autoencoder.
However, existing methods typically use symmetric architectures where the encoder and decoder share the same receptive field size. The authors argue this is fundamentally flawed. Encoders and decoders serve opposite purposes: an encoder needs a broad, global view to capture long-range dependencies and context, while a decoder needs a narrow, localized view to reconstruct sharp textures and fine details. Forcing them to be symmetric either starves the encoder of context or blurs the decoder's outputs.
Additionally, applying these models to real-world problems like meteorology is bottlenecked by a severe shortage of large-scale, open-source datasets.
Methods
The authors decouple the autoencoder's symmetry by proposing the Asymmetric Receptive Field Autoencoder (ARFA), featuring two specialized modules:
- Large Kernel Module (LKM) for the Encoder: To capture long-range spatial dependencies without exploding the parameter count, the encoder utilizes a $9 \times 9$ depthwise separable convolution. Because depthwise convolutions lack cross-channel interaction, the authors add a Multi-Layer Perceptron (MLP) block (two $1 \times 1$ convolutions and a SiLU activation) to recalibrate features. A parallel $3 \times 3$ standard convolution branch acts as a residual connection to preserve local features and stabilize training.
- Small Kernel Module (SKM) for the Decoder: To focus on fine-grained local reconstruction, the decoder employs a multi-head structure with two parallel branches. Each branch uses standard $3 \times 3$ convolutions, layer normalization, and SiLU activations. The outputs are fused via simple addition to maintain sharp local details.
The overall architecture layout can be seen below:
Datasets
The model was evaluated on three datasets, including a new benchmark introduced by the authors:
- Moving-MNIST: A standard synthetic dataset consisting of two digits moving across a $64 \times 64$ grid (10,000 training/testing samples).
- KTH: A real-world dataset of human action videos (128x128 resolution).
- RainBench (New): A newly collected, large-scale radar echo dataset for precipitation forecasting from Yinchuan City, China (spanning April–October 2018 and 2019). It features 31,600 training and 31,600 testing samples at a high $200 \times 200$ resolution. This dataset is uniquely challenging due to the region's low overall rainfall and highly uneven spatiotemporal distribution of precipitation.
Key Insights and Ideas
- Asymmetry is an Optimization: The core "aha moment" is that treating the encoder and decoder as functionally distinct entities yields better performance and lower computational overhead than uniform, symmetric scaling.
- The Receptive Field Sweet Spot: Through ablation studies, the authors found that expanding the encoder's kernel size steadily improves performance up to a point ($9 \times 9$). Pushing it further to $11 \times 11$ actually degrades performance, suggesting an upper bound on useful spatial context for these architectures.
- Mitigating Depthwise Blurring: Large depthwise separable convolutions are great for efficiency but poor for channel interaction. Pairing them with a lightweight $1 \times 1$ MLP effectively solves this limitation, allowing the large kernel to shine.
Key Results
ARFA consistently achieved state-of-the-art (SOTA) metrics against strong baselines like SimVPv2, PredRNN, and ConvLSTM:
- Moving-MNIST: ARFA reduced Mean Squared Error (MSE) to 25.306 and Mean Absolute Error (MAE) to 73.892 (compared to SimVPv2's 27.102 and 78.379).
- KTH: ARFA achieved an MSE of 42.891, visibly outperforming other models which struggled with severe image blurring or failed to reconstruct fast-moving human limbs.
- RainBench: ARFA led the pack with an MSE of 109.880 and MAE of 809.190. While competing models quickly lost image clarity as the prediction time steps increased, ARFA maintained high-fidelity radar echo structures.
Limitations and Future Work
The authors do not explicitly detail architectural limitations or computational bottlenecks for ARFA within the text. However, they note that their new RainBench dataset is inherently difficult to predict due to strong water evaporation and infrequent, uneven rainfall events in the target region. They propose RainBench as an open benchmark for the community to drive future, more robust spatiotemporal forecasting algorithms.
Summary updated