Mechanisms of human dynamic object recognition revealed by sequential deep neural networks

Lynn K. A. Sörensen, Sander M. Bohté, Dorina De Jong, Heleen A. Slagter, H. Steven Scholte

TL;DR

This paper investigates the neural mechanisms behind rapid object recognition in dynamic visual scenes. By developing and testing deep learning models, the researchers found that sequential processing with lateral recurrence and sensory adaptation are crucial for mimicking human performance, especially in fast-paced visual environments. This work offers a computational explanation for how our brains efficiently process changing visual information.

What Problem Does This Paper Address?

Our visual world is constantly in motion, with scenes changing rapidly. Humans are remarkably adept at recognizing objects even when presented with very brief image sequences (as fast as 13 milliseconds per image). However, the underlying computational mechanisms that enable this rapid dynamic object recognition are not well understood. Traditional models often focus on single-image recognition, which doesn't fully capture the challenges of processing continuous streams of visual information where previous images can influence the perception of current ones. This paper aims to bridge this gap by exploring how different computational mechanisms, such as recurrent processing and sensory adaptation, contribute to dynamic object recognition and how these mechanisms relate to human performance.

Methods

The researchers developed a class of deep learning models called BLnext, which are sequential recurrent neural networks designed to process image sequences. These models were compared against several reference models:

  • Single-image models: These models process each image in isolation, with an artificial reset between images. This includes:
  • BLnet: A recurrent single-image model.
  • B: A feedforward single-image model.
  • B-D: A deeper feedforward single-image model.
  • Sequential models: These models, including BLnext, process sequences of images, allowing for temporal integration.

Key computational mechanisms investigated were:

  • Lateral Recurrence: The ability of the network to integrate information across time steps within the sequence.
  • Sensory Adaptation: Mechanisms that suppress the activity of neurons based on recent sensory history, potentially increasing the salience of new information. The study explored both exponential and power-law adaptation.

The models were evaluated on their ability to:

  1. Mimic human performance: By comparing their accuracy on a Rapid Serial Visual Presentation (RSVP) task.
  2. Predict trial-by-trial responses: By assessing how well they could predict human difficulty judgments on individual trials.
  3. Capture temporal dynamics: By examining the relationship between model processing steps and human presentation durations.

Datasets

The models were trained on the EcoSet dataset [38], which is a large-scale image dataset. For evaluating performance and comparing with human behavior, the study used image sequences constructed from the Ecoset test set. The specific stimuli used for the human behavioral study were real-world photographs, with category cues provided to participants.

Key Insights and Ideas

  • Sequential processing is key: Simply processing images one by one, even with recurrent connections, isn't enough. The model needs to integrate information across images in a sequence.
  • Lateral recurrence matters: Models that integrate information sequentially via lateral recurrence were crucial for matching human performance, especially at faster presentation rates.
  • Adaptation accelerates processing: Sensory adaptation significantly improved dynamic recognition performance and allowed models to achieve similar representational quality with fewer processing steps, effectively speeding up the system.
  • Power-law adaptation is more robust: The power-law adaptation mechanism proved more effective than exponential adaptation, particularly for longer sequences, suggesting it better captures the temporal dynamics of adaptation.
  • Temporal correspondence is a strong indicator: Models that showed a realistic temporal link between their processing steps and human presentation durations were better predictors of human behavior.

Key Results

  • Human-like performance: Only sequential, recurrent models (BLnext variants) mirrored human performance on the RSVP task across different presentation durations (13-80 ms/image).
  • Feedforward models are insufficient for dynamics: Single-image models, even feedforward ones, showed high performance but failed to capture the gradual improvement in human sensitivity with longer presentation durations, indicating they don't model the dynamic aspect.
  • Adaptation boosts sensitivity: Models augmented with adaptation showed significantly higher perceptual sensitivity compared to non-adapted models, especially at faster presentation rates.
  • Trial-by-trial prediction: Sequential models, particularly those with adaptation, were better at predicting human trial-by-trial responses, indicating they capture the nuances of perceptual difficulty.
  • Faster temporal link with adaptation: Adaptation allowed models to achieve the same representational quality with fewer processing steps, suggesting a faster temporal link to human performance. For example, a model with power-law adaptation could match human performance at 13 ms/image with only 2 model steps, compared to 6 steps for a non-adapted model.

Limitations and Future Work

  • Performance at longest durations: While sequential models captured human performance up to 80 ms/image, they did not reach human levels at the longest duration (160 ms/image). The authors suggest this might be due to how BLnext was modified (e.g., batch normalization) and that future work could explore better optimization strategies for sequential models.
  • Model vs. human temporal correspondence: While sequential models showed a realistic temporal link, the specific number of model steps per image that best explained human behavior varied across presentation durations. Future work could investigate how to better align these temporal dynamics.
  • Generalizability of adaptation: While adaptation proved beneficial, further research could explore its role in other dynamic visual tasks and its interaction with different network architectures.
  • Optimizing sequential models: The authors suggest that future research should focus on better optimizing sequential models to account for dynamic object recognition across a wider range of timescales.

Summary updated

Abstract

Humans can quickly recognize objects in a dynamically changing world. This ability is showcased by the fact that observers succeed at recognizing objects in rapidly changing image sequences, at up to 13 ms/image. To date, the mechanisms that govern dynamic object recognition remain poorly understood. Here, we developed deep learning models for dynamic recognition and compared different computational mechanisms, contrasting feedforward and recurrent, single-image and sequential processing as well as different forms of adaptation. We found that only models that integrate images sequentially via lateral recurrence mirrored human performance (N = 36) and were predictive of trial-by-trial responses across image durations (13-80 ms/image). Importantly, models with sequential lateral-recurrent integration also captured how human performance changes as a function of image presentation durations, with models processing images for a few time steps capturing human object recognition at shorter presentation durations and models processing images for more time steps capturing human object recognition at longer presentation durations. Furthermore, augmenting such a recurrent model with adaptation markedly improved dynamic recognition performance and accelerated its representational dynamics, thereby predicting human trial-by-trial responses using fewer processing resources. Together, these findings provide new insights into the mechanisms rendering object recognition so fast and effective in a dynamic visual world.
BibTeX
@article{sorensen2023mechanisms,
  author = {Lynn K. A. Sörensen and Sander M. Bohté and Dorina De Jong and Heleen A. Slagter and H. Steven Scholte},
  title = {{Mechanisms of human dynamic object recognition revealed by sequential deep neural networks}},
  journal = {PLOS Computational Biology},
  year = {2023},
  doi = {10.1371/journal.pcbi.1011169},
}
Figure