Reducing connectivity by using cortical modular bands

Julien Vitay, Raphaël Monat, Frédéric Alexandre2004

TL;DR

This paper proposes a biologically inspired, modular connectivity scheme for neural networks that significantly reduces computational complexity while maintaining performance in sensorimotor coordination tasks. By structuring connections into "bands" rather than fully connecting layers, the authors demonstrate a more efficient and robust approach to learning complex associations, particularly in the presence of noise.

What Problem Does This Paper Address?

The paper tackles the fundamental trade-off in neural network design between computational power/complexity and expressiveness. Traditional approaches often fall into two camps: compact, fully connected networks that are efficient for simple tasks but struggle with complexity, or highly expressive but computationally expensive networks. The authors are interested in how to achieve good performance on complex tasks, like sensorimotor coordination, using networks that are both computationally efficient and biologically plausible. They specifically question whether the "complete connectivity" seen in some artificial neural networks is truly necessary, drawing inspiration from the brain's more structured, modular connectivity.

Methods

The authors propose a novel connectivity scheme inspired by the brain's modular structure, specifically "cortical modular bands." Instead of a fully connected layer (like in a standard multi-layer perceptron), they split the associative layer into two distinct maps: an "associative column map" (Ac) and an "associative row map" (Ar).

  • Ac map: Receives input from corresponding columns of the input maps (perception and desired position).
  • Ar map: Receives input from corresponding rows of the input maps.

Crucially, these maps then project to the motor output map in a structured way:

  • Ac map: Projects to corresponding rows of the motor map.
  • Ar map: Projects to corresponding columns of the motor map.

This "banded" connectivity drastically reduces the number of connections from $O(n^4)$ in a fully connected network to $O(n^3)$ in their proposed model, where $n$ is the size of the maps. They implement this using a "value encoding" scheme, where neurons respond to specific intervals of a parameter, and learning is achieved through standard backpropagation. They also explore a variant using Continuous Neural Field Theory (CNFT) for noise suppression.

Datasets

Not discussed in the paper. The experiments are based on a simulated task of centering an object in a camera's view, using synthetic input maps representing object position and desired target position.

Key Insights and Ideas

  • Biological inspiration pays off: The paper strongly suggests that mimicking biological connectivity patterns (modular bands) can lead to significant computational savings without sacrificing performance on complex tasks.
  • Connectivity is key, not just encoding: While "value encoding" is a powerful principle, its effectiveness is amplified when combined with appropriate, biologically plausible connectivity constraints.
  • Robustness to noise: The modular, banded connectivity scheme demonstrates superior robustness to input noise compared to fully connected networks, a critical factor for real-world robotic applications.
  • Gaussian tuning is critical: The performance of both network types is highly sensitive to the width of the Gaussian tuning curves used for information encoding/decoding. There's an optimal range, and this optimum is shifted for the reduced-connectivity network.

Key Results

  • Performance: The proposed partially connected model achieves comparable performance to a fully connected model on a sensorimotor coordination task, with only a slight degradation in performance for perfect inputs.
  • Efficiency: The number of connections is reduced from $3 \times n^4$ (fully connected) to $6 \times n^3$ (partially connected), representing a significant computational saving.
  • Noise Robustness: At noise levels above 10%, the partially connected model becomes significantly more robust than the fully connected model, as shown in Figure 4.
  • Optimal Gaussian Radius: The optimal Gaussian radius for learning is higher for the partially connected network than for the fully connected one.

Limitations and Future Work

  • Real-world validation: The experiments are conducted in simulation. The authors acknowledge that further testing on real robotic platforms is necessary.
  • Generalizability: While the proposed scheme is shown to be effective for sensorimotor coordination, its applicability to other complex tasks would require further investigation.
  • CNFT integration: The CNFT variant showed promise for noise suppression, and further exploration of its integration with the modular connectivity could be beneficial.
  • Further biological constraints: The authors suggest that incorporating additional biological constraints could further enhance performance and efficiency.

Summary updated

Abstract

The way information is represented and processed in a neural network may have important consequences on its computational power and complexity. Basically, information representation refers to distributed or localist encoding and information processing refers to schemes of connectivity that can be complete or minimal. In the past, theoretical and biologically inspired approaches of neural computation have insisted on complementary views (respectively distributed and complete versus localist and minimal) with complementary arguments (complexity versus expressiveness). In this paper, we report experiments on biologically inspired neural networks performing sensorimotor coordination that indicate that a localist and minimal view may have good performances if some connectivity constraints (also coming from biological inspiration) are respected. 1
BibTeX
@misc{vitay2004reducing,
  author = {Julien Vitay and Raphaël Monat and Frédéric Alexandre},
  title = {{Reducing connectivity by using cortical modular bands}},
  year = {2004},
}
Figure