ARFA: An Asymmetric Receptive Field Autoencoder Model for Spatiotemporal Prediction

Wenxuan Zhang, Xuechao Zou, Li Wu, Xiaoying Wang, Jianqiang Huang, Junliang Xing

TL;DR

Most spatiotemporal prediction models rely on symmetric autoencoders, applying identical receptive field sizes to both encoding and decoding steps. This paper introduces ARFA (Asymmetric Receptive Field Autoencoder), which pairs a large-kernel encoder for global context extraction with a small-kernel decoder for local detail reconstruction. ARFA achieves state-of-the-art results across multiple benchmarks and introduces RainBench, a massive new radar echo dataset for precipitation forecasting.

What Problem Does This Paper Address?

Spatiotemporal prediction—predicting future frames in a sequence, such as video frames or weather patterns—requires deep neural networks to master both spatial structures and temporal transitions. The dominant architecture for this task is the autoencoder.

However, existing methods typically use symmetric architectures where the encoder and decoder share the same receptive field size. The authors argue this is fundamentally flawed. Encoders and decoders serve opposite purposes: an encoder needs a broad, global view to capture long-range dependencies and context, while a decoder needs a narrow, localized view to reconstruct sharp textures and fine details. Forcing them to be symmetric either starves the encoder of context or blurs the decoder's outputs.

Additionally, applying these models to real-world problems like meteorology is bottlenecked by a severe shortage of large-scale, open-source datasets.

Methods

The authors decouple the autoencoder's symmetry by proposing the Asymmetric Receptive Field Autoencoder (ARFA), featuring two specialized modules:

  1. Large Kernel Module (LKM) for the Encoder: To capture long-range spatial dependencies without exploding the parameter count, the encoder utilizes a $9 \times 9$ depthwise separable convolution. Because depthwise convolutions lack cross-channel interaction, the authors add a Multi-Layer Perceptron (MLP) block (two $1 \times 1$ convolutions and a SiLU activation) to recalibrate features. A parallel $3 \times 3$ standard convolution branch acts as a residual connection to preserve local features and stabilize training.
  2. Small Kernel Module (SKM) for the Decoder: To focus on fine-grained local reconstruction, the decoder employs a multi-head structure with two parallel branches. Each branch uses standard $3 \times 3$ convolutions, layer normalization, and SiLU activations. The outputs are fused via simple addition to maintain sharp local details.

The overall architecture layout can be seen below:

ARFA Architecture

Datasets

The model was evaluated on three datasets, including a new benchmark introduced by the authors:

  • Moving-MNIST: A standard synthetic dataset consisting of two digits moving across a $64 \times 64$ grid (10,000 training/testing samples).
  • KTH: A real-world dataset of human action videos (128x128 resolution).
  • RainBench (New): A newly collected, large-scale radar echo dataset for precipitation forecasting from Yinchuan City, China (spanning April–October 2018 and 2019). It features 31,600 training and 31,600 testing samples at a high $200 \times 200$ resolution. This dataset is uniquely challenging due to the region's low overall rainfall and highly uneven spatiotemporal distribution of precipitation.

Key Insights and Ideas

  • Asymmetry is an Optimization: The core "aha moment" is that treating the encoder and decoder as functionally distinct entities yields better performance and lower computational overhead than uniform, symmetric scaling.
  • The Receptive Field Sweet Spot: Through ablation studies, the authors found that expanding the encoder's kernel size steadily improves performance up to a point ($9 \times 9$). Pushing it further to $11 \times 11$ actually degrades performance, suggesting an upper bound on useful spatial context for these architectures.
  • Mitigating Depthwise Blurring: Large depthwise separable convolutions are great for efficiency but poor for channel interaction. Pairing them with a lightweight $1 \times 1$ MLP effectively solves this limitation, allowing the large kernel to shine.

Key Results

ARFA consistently achieved state-of-the-art (SOTA) metrics against strong baselines like SimVPv2, PredRNN, and ConvLSTM:

  • Moving-MNIST: ARFA reduced Mean Squared Error (MSE) to 25.306 and Mean Absolute Error (MAE) to 73.892 (compared to SimVPv2's 27.102 and 78.379).
  • KTH: ARFA achieved an MSE of 42.891, visibly outperforming other models which struggled with severe image blurring or failed to reconstruct fast-moving human limbs.
  • RainBench: ARFA led the pack with an MSE of 109.880 and MAE of 809.190. While competing models quickly lost image clarity as the prediction time steps increased, ARFA maintained high-fidelity radar echo structures.
Visual Results on Moving-MNIST and KTH

Limitations and Future Work

The authors do not explicitly detail architectural limitations or computational bottlenecks for ARFA within the text. However, they note that their new RainBench dataset is inherently difficult to predict due to strong water evaporation and infrequent, uneven rainfall events in the target region. They propose RainBench as an open benchmark for the community to drive future, more robust spatiotemporal forecasting algorithms.

Summary updated

Abstract

Spatiotemporal prediction aims to generate future sequences by paradigms learned from historical contexts. It is essential in numerous domains, such as traffic flow prediction and weather forecasting. Recently, research in this field has been predominantly driven by deep neural networks based on autoencoder architectures. However, existing methods commonly adopt autoencoder architectures with identical receptive field sizes. To address this issue, we propose an Asymmetric Receptive Field Autoencoder (ARFA) model, which introduces corresponding sizes of receptive field modules tailored to the distinct functionalities of the encoder and decoder. In the encoder, we present a large kernel module for global spatiotemporal feature extraction. In the decoder, we develop a small kernel module for local spatiotemporal information reconstruction. Experimental results demonstrate that ARFA consistently achieves state-of-the-art performance on popular datasets. Additionally, we construct the RainBench, a large-scale radar echo dataset for precipitation prediction, to address the scarcity of meteorological data in the domain.
BibTeX
@inproceedings{zhang2024arfa,
  author = {Wenxuan Zhang and Xuechao Zou and Li Wu and Xiaoying Wang and Jianqiang Huang and Junliang Xing},
  title = {{ARFA: An Asymmetric Receptive Field Autoencoder Model for Spatiotemporal Prediction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446477},
}
Figure