ARMA Nets: Expanding Receptive Field for Dense Prediction

Jiahao Su, Shiqi Wang, Furong Huang

In "dense prediction" tasks—such as semantic segmentation (labeling every pixel in an image) or video prediction—global context is vital. A model needs to "see" distant parts of an image to accurately understand what is happening locally.

Traditional Convolutional Neural Networks (CNNs) have a limited Effective Receptive Field (ERF). To expand this field and capture global information, researchers typically stack many layers, use large filters, or apply pooling and downsampling. However, these traditional methods come with heavy costs:

  • Deep stacks and large filters drastically increase model complexity and the risk of overfitting.
  • Pooling coarsely aggregates data, losing high-frequency spatial details crucial for pixel-level accuracy.
  • Dilated Convolutions can cause "gridding artifacts" (checkerboard-like patterns) due to gaps in their sampling.
  • Non-local Networks (Attention) are computationally expensive, making them difficult to use in high-resolution layers.

To address this, researchers have introduced ARMA Nets, featuring a novel "plug-and-play" neural network layer that replaces standard convolutions with Autoregressive Moving-Average (ARMA) modules.

The ARMA Solution

The core innovation of this work is the ARMA Layer. A standard convolution is mathematically a "Moving Average" (MA) model, meaning the output depends only on a local window of inputs. The ARMA layer adds an "Autoregressive" (AR) component. In this setup, an output neuron's value depends on both the input and the values of its neighboring output neurons.

By allowing output neurons to influence one another through learnable interconnections, ARMA layers can adaptively expand their effective receptive field to cover an entire image. This achieves global context without the information loss associated with downsampling or the high computational cost of non-local attention.

Comparison of receptive fields: Convolution vs. ARMA vs. Dilated variants.

Overcoming Computation and Stability Hurdles

To make the ARMA layer practical, the authors had to solve two major mathematical and computational hurdles:

  1. Computation: Because output neurons are interdependent, they cannot be calculated sequentially without massive slowdowns. The authors utilized Fast Fourier Transforms (FFT) to solve the underlying equations in the frequency domain, making the layer highly efficient ($O(N \log N)$ complexity) for both forward and backward passes.
  2. Instability: Recursive feedback loops in autoregressive models can "explode," with values trending toward infinity. The authors proposed a Separable ARMA layer and a clever re-parameterization mechanism that mathematically guarantees the model remains stable during training. Without this mechanism, ARMA networks failed to converge; with it, they trained as reliably as standard CNNs.

Key Insights and Performance

The theoretical and empirical evaluations of ARMA Nets revealed several powerful advantages:

  • Infinite Receptive Field with One Parameter: Theoretically, an ARMA layer can have an arbitrarily large receptive field controlled by a single learnable coefficient. This allows the model to automatically "choose" how much global context it needs for a specific task.
  • Filling the Gaps: When combined with dilated convolutions, ARMA layers act as an "anti-aliasing filter." They fill in the spatial information gaps that usually cause gridding artifacts in standard dilated networks.
  • Task-Adaptive Learning: By examining the learned coefficients, the authors found that the model automatically learns large receptive fields for video prediction (where tracking global motion matters) and smaller, more local fields for simpler image tasks.

Benchmark Results

ARMA Nets were evaluated on challenging dense prediction benchmarks, showing significant improvements over strong baselines:

  • Video Prediction (Moving-MNIST-2): An ARMA-LSTM with small $3 \times 3$ kernels outperformed a standard ConvLSTM with larger $5 \times 5$ kernels, achieving a 6.7% improvement in PSNR while using 63.7% fewer parameters.
  • Medical Segmentation (ISIC 2018 Challenge): The ARMA U-Net outperformed both the standard U-Net and the computationally heavy "Non-Local" U-Net across almost all metrics (Accuracy, F1-score, Jaccard Index) for skin lesion segmentation.

Limitations and Future Work

While the ARMA layer is highly efficient in theory, its reliance on FFT-based solvers can make it more complex to implement in custom hardware or specific edge-deployment frameworks compared to standard convolutions.

The authors suggest that future work could explore applying ARMA layers to other domains beyond dense image prediction, such as Graph Neural Networks or Natural Language Processing, where long-range dependencies are equally critical. They also note the potential for misusing dense prediction technologies (e.g., in military object detection) and call for further research into the robustness and responsible deployment of these systems.

Summary updated

Abstract

Global information is essential for dense prediction problems, whose goal is to compute a discrete or continuous label for each pixel in the images. Traditional convolutional layers in neural networks, initially designed for image classification, are restrictive in these problems since the filter size limits their receptive fields. In this work, we propose to replace any traditional convolutional layer with an autoregressive moving-average (ARMA) layer, a novel module with an adjustable receptive field controlled by the learnable autoregressive coefficients. Compared with traditional convolutional layers, our ARMA layer enables explicit interconnections of the output neurons and learns its receptive field by adapting the autoregressive coefficients of the interconnections. ARMA layer is adjustable to different types of tasks: for tasks where global information is crucial, it is capable of learning relatively large autoregressive coefficients to allow for an output neuron's receptive field covering the entire input; for tasks where only local information is required, it can learn small or near zero autoregressive coefficients and automatically reduces to a traditional convolutional layer. We show both theoretically and empirically that the effective receptive field of networks with ARMA layers (named ARMA networks) expands with larger autoregressive coefficients. We also provably solve the instability problem of learning and prediction in the ARMA layer through a re-parameterization mechanism. Additionally, we demonstrate that ARMA networks substantially improve their baselines on challenging dense prediction tasks, including video prediction and semantic segmentation.
BibTeX
@inproceedings{su2020arma,
  author = {Jiahao Su and Shiqi Wang and Furong Huang},
  title = {{ARMA Nets: Expanding Receptive Field for Dense Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=3eVCoz8fnu3},
}
Figure