How can deep learning advance computational modeling of sensory information processing?

Jessica A. F. Thompson, Yoshua Bengio, Elia Formisano, Marc SchönwiesnerarXiv 2018 · arXiv

Introduction

In the burgeoning field of information-based neuroimage analysis, researchers have long sought to decode the brain's internal language by predicting neural activity through computational models. While methods such as Representational Similarity Analysis (RSA) and encoding models have successfully identified what information specific brain regions contain, they often fail to explain the form or the "how" of those representations. This paper argues that task-optimized Deep Neural Networks (DNNs) provide a necessary framework to bridge the gap between biological implementation and computational theory, moving beyond simple content identification toward algorithmic understanding.

The Conceptual Hurdle: Content vs. Form

A central challenge in current neuroimaging analysis is the tendency toward imprecise statistical conclusions. When a model—such as a set of acoustic features—successfully predicts activity in the auditory cortex, it is common to claim the brain "encodes" those features. However, the authors argue this is a misleading simplification.

To illustrate this, consider the difference between Roman numerals (XXXVII) and Arabic numerals (37). Both represent the same content (the number 37), but their form is fundamentally different. The Arabic format is optimized for positional arithmetic, whereas the Roman format is not. In neuroscience, identifying that a brain region contains certain information (content) does not tell us how that information is structured to facilitate specific computations (form). Current methods like RSA and encoding models primarily prove that a brain's representation lives in a similar linear subspace as the model, but they do not prove the brain utilizes the same algorithm.

Re-evaluating Marr’s Levels of Analysis

The authors frame their perspective using Marr’s classic three levels of analysis:

  1. Computational Theory: The ultimate goal or task of the system.
  2. Representation and Algorithm: The format of information and the transformations it undergoes.
  3. Hardware Implementation: The physical realization in neurons or silicon.

Traditional "unlearned" models—static feature sets used in encoding—often skip the algorithmic level, jumping from hardware observations to computational goals. In contrast, DNNs are "goal-driven"; they learn representations specifically to solve a task, such as object recognition or speech processing. This makes DNNs uniquely suited to model the "Representation and Algorithm" level, as they are functionally constrained by the task yet flexible in their internal implementation.

The "Inside-Out" Approach and the Manifold Perspective

Rather than relying solely on bottom-up (data-driven) or top-down (behavior-driven) approaches, the authors advocate for an "Inside-Out" strategy. This involves starting at the level of representation and algorithm. From this perspective, a neural representation is viewed as a "manifold"—the geometric shape of data in a high-dimensional space. DNNs are particularly adept at "disentangling" these manifolds, transforming raw sensory input into structured formats that allow for high-level cognition.

Furthermore, the authors challenge the notion that DNNs are impenetrable "black boxes." They argue that "post-hoc interpretability"—the ability to visualize, probe, and verbalize what a network has learned—actually gives DNNs an advantage over simpler linear models when trying to understand complex sensory processing.

A New Roadmap for Neuroimaging

The paper concludes that the future of the field lies in moving away from mapping single DNN layers to isolated brain regions. Instead, researchers should adopt a network-to-network mapping approach, evaluating entire processing pathways as integrated systems.

Key theoretical results from this framework include:

  • Critique of RSA/Encoding: High correlation in these models does not imply algorithmic identity; it only suggests a shared linear subspace.
  • Intermediate Representations: DNNs demonstrate that intermediate, non-physical representations (those not directly corresponding to external stimuli) are necessary for complex tasks, mirroring the complexity found in the brain's hidden layers.
  • Integration of Brain Graphs: The authors suggest using functional and structural connectivity maps (brain graphs) as templates for DNN architectures. By building artificial networks that mirror the physical "wiring" of the brain, researchers can test how biological constraints give rise to specific representational forms.

Conclusion

By shifting the focus from identifying content to understanding the algorithmic form of representations, deep learning offers a path toward more robust computational models of the brain. While challenges remain regarding biological plausibility and the granularity mismatch between voxels and neurons, the integration of task-optimized DNNs with neuroimaging data represents the most promising route toward explaining how the brain processes sensory information.

Summary updated

Abstract

Deep learning, computational neuroscience, and cognitive science have overlapping goals related to understanding intelligence such that perception and behaviour can be simulated in computational systems. In neuroimaging, machine learning methods have been used to test computational models of sensory information processing. Recently, these model comparison techniques have been used to evaluate deep neural networks (DNNs) as models of sensory information processing. However, the interpretation of such model evaluations is muddied by imprecise statistical conclusions. Here, we make explicit the types of conclusions that can be drawn from these existing model comparison techniques and how these conclusions change when the model in question is a DNN. We discuss how DNNs are amenable to new model comparison techniques that allow for stronger conclusions to be made about the computational mechanisms underlying sensory information processing.
BibTeX
@misc{thompson2018how,
  author = {Jessica A. F. Thompson and Yoshua Bengio and Elia Formisano and Marc Schönwiesner},
  title = {{How can deep learning advance computational modeling of sensory information processing?}},
  year = {2018},
  eprint = {1810.08651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.08651},
}
Figure