How Do Neural Networks Estimate Optical Flow? A Neuropsychology-Inspired Study.

D. B. de Jong, Federico Paredes-Vallés, Guido C. H. E. de CroonIEEE TPAMI 2020 · DOI · OpenReview

Abstract

End-to-end trained convolutional neural networks have led to a breakthrough in optical flow estimation. The most recent advances focus on improving the optical flow estimation by improving the architecture and setting a new benchmark on the publicly available MPI-Sintel dataset. Instead, in this article, we investigate how deep neural networks estimate optical flow. A better understanding of how these networks function is important for (i) assessing their generalization capabilities to unseen inputs, and (ii) suggesting changes to improve their performance. For our investigation, we focus on FlowNetS, as it is the prototype of an encoder-decoder neural network for optical flow estimation. Furthermore, we use a filter identification method that has played a major role in uncovering the motion filters present in animal brains in neuropsychological research. The method shows that the filters in the deepest layer of FlowNetS are sensitive to a variety of motion patterns. Not only do we find translation filters, as demonstrated in animal brains, but thanks to the easier measurements in artificial neural networks, we even unveil dilation, rotation, and occlusion filters. Furthermore, we find similarities in the refinement part of the network and the perceptual filling-in process which occurs in the mammal primary visual cortex.

Introduction

While convolutional neural networks (CNNs) have achieved state-of-the-art performance in optical flow estimation, they are often criticized as "black boxes." Understanding the internal mechanisms of these models is crucial for ensuring their reliability in real-world applications and for guiding future architectural improvements. This study bridges the gap between artificial intelligence and neuroscience by applying established neuropsychological analysis techniques to FlowNetS, a foundational encoder-decoder architecture for motion estimation.

Methodology: Receptive Field Mapping

The researchers treated FlowNetS as a biological subject, employing a "receptive field mapping" approach. The network was presented with synthetic stimuli—specifically, translating plane waves consisting of moving stripes with varying spatial frequencies, orientations, and temporal velocities.

Schematic of the FlowNetS architecture showing the contracting (encoding) and expanding (decoding) parts.

By measuring the activations of neurons in the deepest encoding layer (c6), the authors could profile the spectral response of the network. These responses were then fitted to 3D spatiotemporal Gabor functions, which are standard mathematical models for biological motion sensors. This allowed for a direct comparison between the artificial filters and those found in the mammalian visual cortex.

Illustration of the 3D frequency domain used to fit the Gabor filters.

Key Insights

Convergent Evolution of Motion Filters

The study found that FlowNetS naturally learns motion filters that are remarkably similar to Gabor filters. These filters are mathematically optimal for localizing signals in both space and frequency. The quantitative properties of these learned filters—such as a median orientation bandwidth of 52° and a spatial frequency bandwidth of 1.36 octaves—closely match biological observations in macaques.

Beyond Simple Translation

Unlike most biological studies which focus on translation, the artificial nature of FlowNetS allowed researchers to identify more complex motion primitives. Approximately 15% of the active filters in the c6 layer were more sensitive to dilation (scaling) than translation, and 45% showed a preference for rotation. This suggests that the network develops a comprehensive bank of motion sensors to handle diverse visual scenarios.

Refinement and the "Filling-in" Process

The "expanding part" or decoder of the network appears to function similarly to the perceptual filling-in process observed in the primary visual cortex. It propagates high-confidence motion cues from edges into the interior of objects, effectively regularizing the flow field.

Results and Discussion

The Gabor model proved highly effective, capturing the behavior of the majority of filters with an average error of only 13% on translating waves. However, the study also identified "outlier" filters with complex directional biases or noisy patterns, potentially indicating sensitivity to higher-order features like occlusions.

One significant finding was the "two-frame limit." By analyzing the temporal frequency resolution, the authors concluded that the standard two-frame input severely restricts the network's ability to perceive motion over time. They propose that increasing the temporal depth (i.e., using more frames) could significantly enhance motion perception accuracy.

Conclusion

By looking "under the hood" using tools from neuropsychology, this research demonstrates that artificial neural networks often rediscover biological solutions for complex tasks like motion estimation. These findings not only provide a path toward more interpretable AI but also suggest concrete architectural improvements, such as increasing temporal context, to further advance the field of computer vision.

Summary updated

Abstract

End-to-end trained convolutional neural networks have led to a breakthrough in optical flow estimation. The most recent advances focus on improving the optical flow estimation by improving the architecture and setting a new benchmark on the publicly available MPI-Sintel dataset. Instead, in this article, we investigate how deep neural networks estimate optical flow. A better understanding of how these networks function is important for (i) assessing their generalization capabilities to unseen inputs, and (ii) suggesting changes to improve their performance. For our investigation, we focus on FlowNetS, as it is the prototype of an encoder-decoder neural network for optical flow estimation. Furthermore, we use a filter identification method that has played a major role in uncovering the motion filters present in animal brains in neuropsychological research. The method shows that the filters in the deepest layer of FlowNetS are sensitive to a variety of motion patterns. Not only do we find translation filters, as demonstrated in animal brains, but thanks to the easier measurements in artificial neural networks, we even unveil dilation, rotation, and occlusion filters. Furthermore, we find similarities in the refinement part of the network and the perceptual filling-in process which occurs in the mammal primary visual cortex.
BibTeX
@article{jong2020how,
  author = {D. B. de Jong and Federico Paredes-Vallés and Guido C. H. E. de Croon},
  title = {{How Do Neural Networks Estimate Optical Flow? A Neuropsychology-Inspired Study.}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  doi = {10.1109/tpami.2021.3083538},
}
Figure