Modelling the role of contour integration in visual inference

Salman Khan, Alexander Wong, Bryan TrippbioRxiv 2022 · DOI

TL;DR

This paper embeds a biologically inspired model of contour integration into a deep neural network to explore its role in visual perception. Surprisingly, while the model accurately replicates brain-like responses on synthetic contour tasks, it doesn't significantly improve performance on natural image tasks compared to a simpler feedforward network, suggesting contour integration might not be universally essential for complex visual inference.

What Problem Does This Paper Address?

The human visual system excels at perceiving contours, even in noisy or complex scenes. A key mechanism for this is "contour integration," where the brain enhances neural responses to edges that form a continuous contour. While computational models have shed light on the mechanisms of contour integration in early visual areas like V1, its actual role in high-level visual perception, especially with natural images, remains unclear. This paper aims to bridge this gap by integrating a biologically grounded contour integration model into a task-driven deep neural network and testing its performance on both synthetic and natural image tasks.

Methods

The authors developed a "contour integration (CI) block" based on a biologically plausible circuit model of V1 lateral connections. This block was then integrated into a larger deep neural network architecture, which included an edge extraction front-end and classification layers. The CI block itself models interactions between excitatory and inhibitory neurons within orientation columns, with lateral connections that are learnable. Crucially, the entire network, including the CI block, was trained end-to-end using gradient descent. They also introduced a "feedforward control network" with a similar number of parameters but lacking the recurrent CI block for comparison. A further variant, the "relaxed-positivity-constraint model" (RPCM), was developed to investigate the impact of Dale's principle (separate excitatory and inhibitory neurons) by allowing weights to be positive or negative.

Datasets

  • Synthetic contour fragments dataset: Used for training and evaluating the model on tasks mimicking classic contour integration experiments. This dataset featured stimuli with aligned line segments forming contours against a background of random orientations, varying in length and spacing.
  • Barcelona Images for Perceptual Edge Detection (BIPED) dataset: Used for training and evaluating edge detection in natural images. This dataset contains image-edge map pairs.
  • Custom natural image datasets: Created for specific tasks like distinguishing connected vs. unconnected contours and analyzing the effect of fragment spacing in natural scenes.

Key Insights and Ideas

  • Biological realism emerges from task optimization: When trained on synthetic contour tasks, the CI model learned lateral connection patterns and neural responses that closely mirrored biological data, even without explicit hard-coding of these patterns. This suggests that these "biologically realistic" features are indeed optimal for this specific task.
  • Contour integration's role in natural images is nuanced: While the CI model showed some benefits (e.g., enhancing weak edges), it did not consistently outperform a simpler feedforward network on natural image tasks like edge detection or contour tracing. This challenges the assumption that contour integration is a universally critical component for all visual inference tasks.
  • Dale's principle might be a performance bottleneck: The RPCM variant, which relaxed the strict separation of excitatory and inhibitory neurons, consistently outperformed both the original CI model and the feedforward control network across most tasks. This suggests that while Dale's principle might contribute to biological realism, it may not be optimal for task performance in these specific network architectures.
  • Generalization vs. performance trade-off: The CI model showed better generalization to out-of-distribution stimuli in some cases, even when its overall performance was lower than the control.

Key Results

  • Synthetic contour tasks: The CI model closely matched behavioral and neurophysiological data from monkey experiments regarding contour length and fragment spacing effects on neural gains. It outperformed the control network by ~11% on validation scores for contour fragments.
  • Edge detection in natural images: The CI model and the control network performed similarly, with validation IoU scores around 0.45. The CI model showed a slight tendency to enhance weaker edges but this didn't significantly improve overall detection performance.
  • Contour tracing in natural images: The CI model performed worse (~5% lower validation accuracy) than the control network on this task. However, it generalized better to stimuli with varying fragment spacing.
  • RPCM variant: Outperformed the model and control on most tasks, achieving higher IoU scores and accuracies, particularly on the contour tracing task (~13% better than the model).

Limitations and Future Work

  • Limited impact on natural images: The authors acknowledge that the CI mechanism did not consistently improve performance on natural image tasks, suggesting that either the model is missing other crucial biological mechanisms or that contour integration's role in these tasks is less critical than initially hypothesized.
  • Dale's principle: The RPCM variant's superior performance suggests that the strict adherence to Dale's principle in the main model might be a limitation for task optimization. Future work could explore how to reconcile biological realism with optimal task performance, perhaps through different learning rules or network architectures.
  • Ethological relevance: The authors suggest that future work should investigate contour integration in tasks with higher ethological relevance, moving beyond simple edge detection or contour tracing.
  • Functional equivalence: The appendix hints at potential functional equivalence between the model and RPCM under certain conditions, suggesting that transient dynamics and optimization challenges might explain performance differences. Further investigation into these dynamics could be fruitful.

Summary updated

Abstract

Abstract Under difficult viewing conditions, the brain’s visual system uses a variety of recurrent modulatory mechanisms to augment feed-forward processing. One resulting phenomenon is contour integration, which occurs in the primary visual (V1) cortex and strengthens neural responses to edges if they belong to a larger smooth contour. Computational models have contributed to an understanding of the circuit mechanisms of contour integration, but less is known about its role in visual perception. To address this gap, we embedded a biologically grounded model of contour integration in a task-driven artificial neural network, and trained it using a gradient-descent variant. We used this model to explore how brain-like contour integration may be optimized for high-level visual objectives as well as its potential roles in perception. When the model was trained to detect contours in a background of random edges, a task commonly used to examine contour integration in the brain, it closely mirrored the brain in terms of behavior, neural responses, and lateral connection patterns. When trained on natural images, the model enhanced weaker contours and distinguished whether two points lay on the same vs. different contours. The model learnt robust features that generalized well to out-of-training-distribution stimuli. Surprisingly, and in contrast with the synthetic task, a parameter-matched control network without recurrence performed the same or better than the model on the natural-image tasks. Thus a contour integration mechanism is not essential to perform these more naturalistic contour-related tasks. Finally, the best performance in all tasks was achieved by a modified contour integration model that did not distinguish between excitatory and inhibitory neurons. Author summary Deep networks are machine-learning systems that consist of interconnected neuron-like elements. More than other kinds of artificial system, they rival human information processing in a variety of tasks. These structural and functional parallels have raised interest in using deep networks as simplified models of the brain, to better understand of brain function. For example, incorporating additional biological phenomena into deep networks may help to clarify how they affect brain function. In this direction, we adapted a deep network to incorporate a model of visual contour integration, a process in the brain that makes contours appear more visually prominent. We found that suitable training led this model to behave much like the corresponding brain circuits. We then investigated potential roles of the contour integration mechanism in processing of natural images, an important question that has been difficult to answer. The results were not straightforward. For example, the contour integration mechanism actually impaired the network’s ability to tell whether two points lay on the same contour or not, but improved the network’s ability to generalize this skill to a different group of images. Overall, this approach has raised more sophisticated questions about the role of contour integration in natural vision.
BibTeX
@misc{khan2022modelling,
  author = {Salman Khan and Alexander Wong and Bryan Tripp},
  title = {{Modelling the role of contour integration in visual inference}},
  year = {2022},
  doi = {10.1101/2022.10.28.514169},
  howpublished = {bioRxiv},
}
Figure