ARMA Nets: Expanding Receptive Field for Dense Prediction

Jiahao Su, Shiqi Wang, Furong Huang

Abstract

Global information is essential for dense prediction problems, whose goal is to compute a discrete or continuous label for each pixel in the images. Traditional convolutional layers in neural networks, initially designed for image classification, are restrictive in these problems since the filter size limits their receptive fields. In this work, we propose to replace any traditional convolutional layer with an autoregressive moving-average (ARMA) layer, a novel module with an adjustable receptive field controlled by the learnable autoregressive coefficients. Compared with traditional convolutional layers, our ARMA layer enables explicit interconnections of the output neurons and learns its receptive field by adapting the autoregressive coefficients of the interconnections. ARMA layer is adjustable to different types of tasks: for tasks where global information is crucial, it is capable of learning relatively large autoregressive coefficients to allow for an output neuron's receptive field covering the entire input; for tasks where only local information is required, it can learn small or near zero autoregressive coefficients and automatically reduces to a traditional convolutional layer. We show both theoretically and empirically that the effective receptive field of networks with ARMA layers (named as ARMA networks) expands with larger autoregressive coefficients. We also provably solve the instability problem of learning and prediction in the ARMA layer through a re-parameterization mechanism. Additionally, we demonstrate that ARMA networks substantially improve their baselines on challenging dense prediction tasks including video prediction and semantic segmentation.

Introduction

In dense prediction tasks such as semantic segmentation, depth estimation, or video prediction, the objective is to assign a label or value to every pixel in an input image. A critical requirement for high performance in these tasks is the ability to capture global context. For instance, classifying a pixel on the edge of an object often requires understanding the global structure of that object.

Standard Convolutional Neural Networks (CNNs) are limited by their Effective Receptive Field (ERF). While stacking layers or using pooling can increase the ERF, these methods often lead to a loss of spatial resolution. Dilated convolutions offer an alternative but can introduce "gridding artifacts" due to the sparse sampling of the input. More complex solutions like non-local attention modules provide global context but suffer from high computational and memory costs. ARMA Nets address these challenges by introducing a flexible, efficient "plug-and-play" layer that can expand the ERF up to the entire image without sacrificing resolution.

The ARMA Layer

The core contribution of this work is the ARMA layer, which generalizes the standard convolutional layer. A traditional convolution can be viewed as a "Moving-Average" (MA) process, where the output $Y$ is a weighted sum of the input $X$ ($Y = W * X$). The ARMA layer introduces an "Autoregressive" (AR) component, allowing output neurons to interconnect ($A * Y = W * X$).

Comparison of standard convolution (a) and the ARMA layer (b) in 1D. The ARMA layer introduces horizontal interconnections between output neurons.

Efficient Computation and Stability

Solving the AR component in the spatial domain is computationally intensive because of the recursive dependencies between output neurons. The authors circumvent this by performing the operation in the frequency domain using Fast Fourier Transforms (FFT). In the frequency domain, the convolution becomes an element-wise multiplication, and the autoregressive component becomes an element-wise division, making the computation highly efficient ($O(I^2 \log I)$ for an $I \times I$ image).

To prevent the recursive connections from "exploding" (instability), the authors introduce a re-parameterization mechanism. By using rotations and tanh functions, the learnable autoregressive coefficients are constrained within a stable range, ensuring the network converges reliably during training.

Experimental Evaluation

The effectiveness of ARMA Nets was evaluated across several challenging benchmarks:

  1. Video Prediction (Moving-MNIST-2): In scenarios with high-speed motion (3x speed), the ARMA-LSTM outperformed standard ConvLSTM baselines by 6.7% in PSNR while requiring 63.7% fewer parameters. This demonstrates the layer's ability to capture long-range temporal and spatial dependencies.
  2. Medical Image Segmentation (ISIC 2018): The ARMA U-Net achieved a Jaccard Index of 0.780, surpassing both the standard U-Net (0.754) and the Non-Local U-Net (0.741), highlighting its utility in dense pixel-wise classification.
  3. Image Classification (CIFAR-10/100): These tests showed that the ARMA layer is adaptive. In tasks where global context is less critical due to existing pooling layers, the AR coefficients learned to be near zero, effectively reducing the layer to a standard convolution.

Key Insights

The ARMA layer provides several unique advantages over traditional architectures:

  • Flexible Receptive Field: The ERF is controlled by learnable parameters. It can range from a local 3x3 patch to the entire image frame.
  • Anti-Aliasing: When paired with dilated convolutions, the ARMA component acts as a learnable filter that fills the gaps in sparse sampling, mitigating gridding artifacts.
  • Efficiency: Compared to non-local blocks ($O(I^4)$), ARMA layers are significantly more scalable for high-resolution images.
Receptive field comparison: (a) standard convolution, (b) ARMA, (c) dilated convolution, and (d) dilated ARMA. Note how the ARMA component (b, d) fills the receptive field compared to the sparse or local standard versions.

Conclusion

ARMA Nets offer a powerful and efficient way to expand the receptive field for dense prediction tasks. By integrating autoregressive interconnections and ensuring stability through re-parameterization, these layers provide a flexible alternative to standard convolutions and expensive attention mechanisms. Future work may explore non-linear recursions or the application of ARMA layers to 3D data such as video volumes and medical scans.

Summary updated

Abstract

Global information is essential for dense prediction problems, whose goal is to compute a discrete or continuous label for each pixel in the images. Traditional convolutional layers in neural networks, initially designed for image classification, are restrictive in these problems since the filter size limits their receptive fields. In this work, we propose to replace any traditional convolutional layer with an autoregressive moving-average (ARMA) layer, a novel module with an adjustable receptive field controlled by the learnable autoregressive coefficients. Compared with traditional convolutional layers, our ARMA layer enables explicit interconnections of the output neurons and learns its receptive field by adapting the autoregressive coefficients of the interconnections. ARMA layer is adjustable to different types of tasks: for tasks where global information is crucial, it is capable of learning relatively large autoregressive coefficients to allow for an output neuron's receptive field covering the entire input; for tasks where only local information is required, it can learn small or near zero autoregressive coefficients and automatically reduces to a traditional convolutional layer. We show both theoretically and empirically that the effective receptive field of networks with ARMA layers (named as ARMA networks) expands with larger autoregressive coefficients. We also provably solve the instability problem of learning and prediction in the ARMA layer through a re-parameterization mechanism. Additionally, we demonstrate that ARMA networks substantially improve their baselines on challenging dense prediction tasks including video prediction and semantic segmentation.
BibTeX
@inproceedings{su2020arma,
  author = {Jiahao Su and Shiqi Wang and Furong Huang},
  title = {{ARMA Nets: Expanding Receptive Field for Dense Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=3eVCoz8fnu3},
}
Figure