Learning long-range spatial dependencies with horizontal gated-recurrent units

Drew Linsley, Junkyung Kim, Vijay Veerabadran, Thomas Serre

Abstract

Progress in deep learning has spawned great successes in many engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural networks, are now approaching -- and sometimes even surpassing -- human accuracy on a variety of visual recognition tasks. Here, however, we show that these neural networks and their recent extensions struggle in recognition tasks where co-dependent visual features must be detected over long spatial ranges. We introduce the horizontal gated-recurrent unit (hGRU) to learn intrinsic horizontal connections -- both within and across feature columns. We demonstrate that a single hGRU layer matches or outperforms all tested feedforward hierarchical baselines including state-of-the-art architectures which have orders of magnitude more free parameters. We further discuss the biological plausibility of the hGRU in comparison to anatomical data from the visual cortex as well as human behavioral data on a classic contour detection task.

Introduction

While Convolutional Neural Networks (CNNs) have revolutionized computer vision, their architecture is fundamentally limited by the "receptive field" problem. To integrate information across a large image, a feedforward CNN must be extremely deep, stacking many layers so that the top-level neurons eventually "see" the whole picture. This approach is often inefficient for handling long-range spatial dependencies—tasks where the identity of a pixel depends on other pixels far away.

Overview of spatial dependency tasks including natural image contours (a), the Pathfinder challenge (b), and map navigation (c).

The authors demonstrate that even state-of-the-art models like ResNets struggle with simple geometric tasks that humans find trivial, suggesting that feedforward depth is a poor substitute for the recurrent, horizontal integration found in biological brains.

Methods: The horizontal Gated Recurrent Unit (hGRU)

The authors propose the hGRU, a novel recurrent neural network (RNN) cell designed for spatial data. It is based on a computational model of the visual cortex, where neurons in the same layer have "horizontal connections" that facilitate contour grouping.

The hGRU extends the standard GRU by adding spatial kernels that compute inhibitory and excitatory interactions within a single layer. Processing occurs in two stages:

  1. Inhibitory Stage: Suppresses noise and clutter.
  2. Excitatory Stage: Facilitates the "growth" of a path or contour.

To maintain parameter efficiency, the model uses symmetric weights between channels, nearly halving the number of learnable parameters compared to a standard convolutional layer.

Experimental Evaluation

The model was evaluated on several benchmarks designed to test spatial reasoning:

  1. Pathfinder Challenge: A synthetic benchmark consisting of 150x150 images containing "paddles" (small oriented lines). The task is binary classification: are two white circles connected by a single path of paddles?
  2. BSDS500: A standard natural image dataset for contour detection.
  3. Synthetic Contours Dataset (SCD): Used to compare model performance against human psychophysical data.
The Pathfinder challenge requires models to determine if two dots are connected by a single winding path of oriented segments.

Key Results

  • Parameter Efficiency: A single hGRU layer outperformed a ResNet-152 on the Pathfinder task while using ~1000x fewer parameters.
  • Superiority over Baselines: While standard CNNs (ResNet-18, ResNet-50) failed to perform better than chance on the longest paths, the hGRU maintained near-perfect accuracy.
  • Biological Correlation: The model’s performance on contour detection tasks showed a 0.947 correlation with human behavioral data, suggesting it captures fundamental aspects of biological vision.

Discussion

The hGRU demonstrates that recurrence acts as adaptive depth. Instead of requiring hundreds of layers to expand a receptive field, a single hGRU layer can "walk" along a path over several timesteps, allowing the effective receptive field to grow dynamically based on the task. Furthermore, when trained on natural images, the hGRU's learned kernels automatically resemble the "association fields" found in the human visual cortex.

While parameter-efficient, recurrent models can be slower on current hardware due to sequential computation. Future work will explore integrating hGRU modules into hybrid architectures to combine feedforward feature extraction with recurrent spatial reasoning.

Summary updated

Abstract

Progress in deep learning has spawned great successes in many engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural networks, are now approaching -- and sometimes even surpassing -- human accuracy on a variety of visual recognition tasks. Here, however, we show that these neural networks and their recent extensions struggle in recognition tasks where co-dependent visual features must be detected over long spatial ranges. We introduce the horizontal gated-recurrent unit (hGRU) to learn intrinsic horizontal connections -- both within and across feature columns. We demonstrate that a single hGRU layer matches or outperforms all tested feedforward hierarchical baselines including state-of-the-art architectures which have orders of magnitude more free parameters. We further discuss the biological plausibility of the hGRU in comparison to anatomical data from the visual cortex as well as human behavioral data on a classic contour detection task.
BibTeX
@inproceedings{linsley2018learning,
  author = {Drew Linsley and Junkyung Kim and Vijay Veerabadran and Thomas Serre},
  title = {{Learning long-range spatial dependencies with horizontal gated-recurrent units}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=SJWP1tZubS},
}
Figure