Recurrent processing improves occluded object recognition and gives rise to perceptual hysteresis

Markus Ernst, Thomas Burwick, Jochen TrieschJournal of Vision 2021 · DOI

TL;DR

This paper demonstrates that recurrent neural networks significantly outperform feedforward networks in recognizing partially occluded objects, especially in challenging 3D stereoscopic scenarios. The authors introduce new datasets and show that recurrent connections enable networks to "discount" occluders over time and exhibit perceptual hysteresis, mimicking human visual perception.

What Problem Does This Paper Address?

For decades, object recognition in both biological and artificial systems has largely been modeled as a feedforward process, where information flows in one direction from input to output. This view is supported by fast human reaction times and the success of deep feedforward neural networks. However, real-world vision often involves incomplete or ambiguous information, such as when objects are partially hidden. This paper investigates whether recurrent connections, which allow information to loop back within a network, are crucial for robust object recognition, particularly in these challenging occluded scenarios, and if they can explain perceptual phenomena like hysteresis.

Methods

The authors compare seven different neural network architectures: purely feedforward models (B, B-F, B-K, B-D) and recurrent models incorporating bottom-up, lateral, and top-down connections (BT, BL, BLT). To ensure fair comparison, they match parameters and introduce control feedforward models (B-K, B-F, B-D) with architectural variations (larger kernels, more filters, deeper layers) to match the complexity of recurrent models. They use truncated backpropagation through time to train recurrent networks. For analysis, they employ Class Activation Mapping (CAM) to visualize what image regions the network focuses on and t-SNE to visualize internal representations. Perceptual hysteresis is investigated by presenting morphing sequences of images to the networks.

Datasets

The paper introduces OSCAR v2, an enhanced version of their stereoscopic occluded object dataset. It includes three sub-datasets:

  • OS-MNIST: Handwritten digits with varying occlusion.
  • OS-fMNIST: Fashion-MNIST items, offering more complex textures and shapes.
  • OS-YCB: 3D objects from the YCB object set, providing a more realistic scenario.

OSCAR v2 features improvements like uncentered objects for translational invariance, scaled objects for perspective, and stereoscopic image pairs. This dataset aims to bridge the gap between simplified digit recognition tasks and complex real-world 3D object recognition.

Key Insights and Ideas

  • Recurrence is Key for Occlusion: The most significant finding is that recurrent architectures consistently and significantly outperform their feedforward counterparts when dealing with occluded objects, especially in stereoscopic vision.
  • Discounting Occluders: Analysis of network activity over time shows that recurrent networks progressively reduce the influence of occluding elements, effectively "discounting" them to focus on the target object.
  • Perceptual Hysteresis: Recurrent networks exhibit perceptual hysteresis, meaning their classification decisions depend not only on the current input but also on the network's previous state. This phenomenon, observed in human perception, suggests recurrent networks are learning a form of temporal memory.
  • Stereoscopic Advantage: Recurrent networks show even greater performance gains with stereoscopic input, suggesting they effectively leverage the additional depth and viewpoint information provided by binocular vision to resolve occlusions.
  • Implicit Memory: The hysteresis effect implies that recurrent networks develop an internal "memory" of past stimuli, which influences current perception, mirroring biological visual processing.

Key Results

  • Recurrent architectures achieved significantly lower error rates across all OSCAR v2 datasets compared to parameter-matched feedforward models. For example, in OS-fMNIST stereo input, recurrent models achieved error rates as low as 0.064%, while the best feedforward model (B-D) reached 0.072%.
  • The deeper feedforward model (B-D) showed improved performance over simpler feedforward models but still lagged behind most recurrent architectures, especially in challenging random-position scenarios.
  • CAM analysis revealed that recurrent networks learn to focus their "attention" on the target object over time, with the importance assigned to occluders diminishing.
  • Hysteresis was observed in recurrent networks when presented with morphing image sequences, with decision boundaries shifting depending on the direction of the sequence (forward vs. backward).

Limitations and Future Work

  • Control Models: While the authors introduce several feedforward control models, they acknowledge the challenge of defining a perfectly matched control, especially for complex recurrent architectures.
  • Generalizability: The authors note that while their results are consistent across different datasets, the performance gains might depend on the specific training regime.
  • Future Work:
    • Investigate the role of lateral connections in preserving information.
  • Explore how recurrent networks might learn "border ownership" mechanisms to distinguish targets from occluders.
  • Analyze the potential for recurrent networks to be more robust against adversarial attacks.
  • Further explore the mechanisms and functions of different forms of hysteresis (positive and negative).
  • Investigate the impact of recurrent processing on rapid serial visual recognition and visual masking.
  • Explore how networks might adapt to learn specific disparities for better occlusion handling in stereoscopic vision.

Summary updated

Abstract

Over the past decades, object recognition has been predominantly studied and modelled as a feedforward process. This notion was supported by the fast response times in psychophysical and neurophysiological experiments and the recent success of deep feedforward neural networks for object recognition. Recently, however, this prevalent view has shifted and recurrent connectivity in the brain is now believed to contribute significantly to object recognition - especially under challenging conditions, including the recognition of partially occluded objects. Moreover, recurrent dynamics might be the key to understanding perceptual phenomena such as perceptual hysteresis. In this work we investigate if and how artificial neural networks can benefit from recurrent connections. We systematically compare architectures comprised of bottom-up, lateral, and top-down connections. To evaluate the impact of recurrent connections for occluded object recognition, we introduce three stereoscopic occluded object datasets, which span the range from classifying partially occluded hand-written digits to recognizing three-dimensional objects. We find that recurrent architectures perform significantly better than parameter-matched feedforward models. An analysis of the hidden representation of the models suggests that occluders are progressively discounted in later time steps of processing. We demonstrate that feedback can correct the initial misclassifications over time and that the recurrent dynamics lead to perceptual hysteresis. Overall, our results emphasize the importance of recurrent feedback for object recognition in difficult situations.
BibTeX
@article{ernst2021recurrent,
  author = {Markus Ernst and Thomas Burwick and Jochen Triesch},
  title = {{Recurrent processing improves occluded object recognition and gives rise to perceptual hysteresis}},
  journal = {Journal of Vision},
  year = {2021},
  doi = {10.1167/jov.21.13.6},
}
Figure