Connectome Mapping: Shape-Memory Network via Interpretation of Contextual Semantic Information

Kyungsu Lee, Haeyun Lee, Jae Youn HwangICLR 2025 · OpenReview

Introduction

Despite the massive success of Deep Learning in computer vision, conventional Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) remain largely "static." Once trained, their internal connections are fixed, regardless of whether they are processing a clear highway or a cluttered forest.

This static nature leads to a "Semantic Domain Gap." Pixel-level classification often fails because models do not explicitly utilize contextual cues—such as the spatial relationship between the sky and the road, or the proportional area cars occupy in a street scene. In contrast, the human brain excels at using this "contextual semantic information" to guide neural signaling through a dynamic connectome. Artificial networks have historically struggled to emulate this dynamic, synaptic gating.

To bridge this gap, researchers have introduced the Shape-Memory Network (SMN), a brain-inspired architecture that "re-wires" itself on the fly to better understand the context of an image. By using "conditional neurons" that activate or deactivate based on semantic content—much like synapses in the human brain—the SMN achieves state-of-the-art performance in semantic segmentation by adapting its structure during inference.

Methods: Emulating the Connectome

The SMN introduces several novel components designed to mimic biological neurotransmission and dynamically adapt to input data.

SMN Architecture and Inference Pipeline
  1. The Conditional Neuron: Unlike a standard artificial neuron, this unit receives a "control signal" that can effectively turn the neuron on or off. This mechanism mimics the electrochemical processes in the brain where synapses are dynamically activated or deactivated based on stimuli.
  2. Density Regression: The network performs a pretext task where it predicts the "density" (proportional area) of various object classes within the image. This global understanding of what is in the image serves as the control signal for the conditional neurons.
  3. Entropy-Map Reconstruction: To enable the model to adapt at test time without requiring human-annotated labels, the authors developed an algorithm that reconstructs an "entropy map" using Class Activation Maps (CAMs).
  4. Test-Time Adaptation (TTA): During the inference phase, the SMN performs a quick, self-supervised re-optimization. It adjusts a small set of parameters (the control matrix) to "recall" the optimal architecture for the specific semantic context of the input image.

Key Architectural Insights

  • Structural Mutation over Weight Tuning: Most adaptation methods merely nudge network weights. The SMN actually changes its topology at test time. By activating or deactivating connections, it essentially becomes a specialized model for every unique image it processes.
  • Context as a Control Signal: The core innovation is using object density as a proxy for context. If the model recognizes a high-density urban environment, the control signal triggers a network structure optimized for that specific geometry.
  • Self-Activation: To solve the "sparsity problem"—where too many deactivated neurons might result in lost information—the architecture includes a self-activation mechanism. This ensures that highly informative features can still pass through the network even if the global control signal is weak.

Key Results and Benchmarks

The SMN was rigorously tested across a diverse array of benchmarks to prove its versatility, including Scene Parsing (ADE20K, Youtube-VOS), Autonomous Driving (BDD100K), Aerial Imagery (Inria, LoveDA), and Synthetic Data (GTA5).

Performance Comparison Across Benchmarks

The SMN consistently outperformed baseline models (like U-Net and SegFormer) and state-of-the-art architectures (like InternImage and BEiT-3):

  • Performance Boost: On average, the SMN provided a 6.36% improvement in Intersection over Union (IoU) over compatible networks, with maximum improvements reaching 13.15% in certain complex scenarios.
  • Efficiency: Despite the test-time optimization, the model remains highly practical. It achieves 32.8 FPS on an M1 Max processor, making it viable for real-world, real-time applications.
  • Robustness: Qualitative visual maps demonstrate that the SMN is significantly better at defining sharp boundaries and identifying small objects compared to static models.

Limitations and Future Work

While the results highlight the immense potential of connectome emulation, the authors note a few areas for future refinement:

  • Hyperparameter Sensitivity: Parameters such as the learning rate for TTA and the threshold for neuron activation were not exhaustively searched in this study. Optimizing these could yield even higher performance.
  • Computational Overhead: Although 32.8 FPS is efficient, the TTA process inherently adds complexity. Future work will explore "lightweight TTA" variants using parameter pruning and quantization.
  • Task Expansion: The current research focuses heavily on semantic segmentation. However, preliminary tests on object detection (using the COCO dataset) suggest that the "Shape-Memory" principle can be generalized to a wide variety of computer vision tasks.

Summary updated

Abstract

Contextual semantic information plays a pivotal role in the brain's visual interpretation of the surrounding environment. When processing visual information, electrical signals within synapses facilitate the dynamic activation and deactivation of synaptic connections, guided by the contextual semantic information associated with different objects. In the realm of Artificial Intelligence (AI), neural networks have emerged as powerful tools to emulate complex signaling systems, enabling tasks such as classification and segmentation by understanding visual information. However, conventional neural networks have limitations in simulating the conditional activation and deactivation of synapses, collectively known as the connectome, a comprehensive map of neural connections in the brain. Additionally, the pixel-wise inference mechanism of conventional neural networks failed to account for the explicit utilization of contextual semantic information in the prediction process. To overcome these limitations, we developed a novel neural network, dubbed the Shape Memory Network (SMN), which excels in two key areas: (1) faithfully emulating the intricate mechanism of the brain's connectome, and (2) explicitly incorporating contextual semantic information during the inference process. The SMN memorizes the structure suitable for contextual semantic information and leverages this structure at the inference phase. The structural transformation emulates the conditional activation and deactivation of synaptic connections within the connectome. Rigorous experimentation carried out across a range of semantic segmentation benchmarks demonstrated the outstanding performance of the SMN, highlighting its superiority and effectiveness. Furthermore, our pioneering network on connectome emulation reveals the immense potential of the SMN for next-generation neural networks.
BibTeX
@inproceedings{lee2025connectome,
  author = {Kyungsu Lee and Haeyun Lee and Jae Youn Hwang},
  title = {{Connectome Mapping: Shape-Memory Network via Interpretation of Contextual Semantic Information}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=PZYr22zFyE},
}
Figure