Multi-dimensional Recurrent Neural Networks

Alex Graves, Santiago Fernández, Jürgen Schmidhuber

Introduction

Recurrent Neural Networks (RNNs) have long been the standard for modeling sequential data due to their ability to maintain internal state over time. However, their application to multi-dimensional data, such as images or video, has historically been limited. Standard RNNs process data in a single dimension, often requiring multi-dimensional inputs to be flattened into 1D vectors, a process that discards vital spatial hierarchies. This paper introduces Multi-Dimensional Recurrent Neural Networks (MDRNNs), a framework that extends the advantages of RNNs—specifically their ability to model long-range dependencies—to data with arbitrary dimensions.

The Challenge of Spatial Context

While Convolutional Neural Networks (CNNs) are highly effective for image processing, they possess inherent limitations. CNNs typically rely on fixed-size kernels, meaning their receptive field is local at each layer. To capture global context, many layers must be stacked. Furthermore, CNNs can be sensitive to spatial distortions like warping unless explicitly augmented during training. Other approaches, such as Multi-Dimensional Hidden Markov Models (HMMs), suffer from exponential computational growth as dimensionality increases.

Methodology: The MDRNN Architecture

The core innovation of the MDRNN is the generalization of the recurrence relation. In a 1D RNN, the hidden state at step $t$ depends on the input at $t$ and the hidden state at $t-1$. In an MDRNN, the hidden state at coordinates $(i, j, \dots)$ depends on the input at those coordinates and the hidden states from the immediately preceding steps in all dimensions.

Ordered Processing and Multi-Directionality

To maintain causality during the forward pass, the network must process data in a specific order such that all required "previous" states are computed before they are needed. For a 2D image, this involves a scan that progresses through the dimensions sequentially.

Scanning order for 2D MDRNN processing, ensuring all previous states are computed before reaching point (i,j).

However, a single scan only provides context from one direction (e.g., the "past" relative to the scan). To provide every point with global context from the entire input, MDRNNs utilize $2^n$ separate hidden layers for an $n$-dimensional input. In 2D, four layers are used, each starting from a different corner of the image. This ensures that every pixel receives information from every other pixel in the image.

Multi-directional context provided by four hidden layers in 2D, allowing the network to incorporate information from all quadrants relative to point (i,j).

Multi-Dimensional LSTM (MD-LSTM)

To mitigate the vanishing gradient problem in higher dimensions, the authors extend the Long Short-Term Memory (LSTM) architecture. The MD-LSTM introduces multiple "forget gates"—one for each incoming dimension. This allows the memory cell to independently decide which information to retain or discard from each spatial direction, providing a sophisticated mechanism for filtering multi-dimensional context.

Experimental Evaluation

The authors tested MDRNNs on tasks requiring both local precision and global context.

Texture Segmentation

Using the Air Freight database, which consists of ray-traced images with complex textures and reflections, the MDRNN was tasked with pixel-wise classification. It achieved a pixel classification error rate of 7.3%, demonstrating an ability to distinguish textures that appear similar locally but differ based on broader spatial context.

Robustness to Warping: MNIST vs. Warped MNIST

A key finding was the model's robustness to elastic deformations. On the standard MNIST dataset, the MDRNN performed comparably to a CNN (1.1% vs 0.9% error). However, on "Warped MNIST"—where images were subjected to random twisting and stretching—the MDRNN significantly outperformed the CNN, achieving a 6.8% error rate compared to the CNN's 11.3%. This suggests that the recurrent structure allows the network to dynamically adapt to spatial distortions in a way that fixed convolutional filters cannot.

Conclusion

MDRNNs provide a powerful alternative to CNNs for multi-dimensional data, offering inherent robustness to warping and the ability to capture global context within a single layer. While the computational constant is higher due to the multiple directional layers, the complexity remains linear $O(N)$ with respect to the number of pixels. This architecture opens the door for applying recurrent modeling to complex 3D and 4D datasets in fields like medical imaging and video analysis.

Summary updated

BibTeX
@inproceedings{graves2007multi,
  author = {Alex Graves and Santiago Fernández and Jürgen Schmidhuber},
  title = {{Multi-dimensional Recurrent Neural Networks}},
  booktitle = {ICANN},
  year = {2007},
  doi = {10.1007/978-3-540-74690-4\_56},
}
Figure