Towards a Deep Network Architecture for Structured Smoothness

Haroun Habeeb, Oluwasanmi KoyejoICLR 2020 · OpenReview

Introduction

In deep learning, a model's "inductive bias"—the set of assumptions it makes about the underlying data—dictates its architecture and ultimate success. Convolutional Neural Networks (CNNs) excel in computer vision because they assume translation invariance: a feature like a cat is recognized regardless of where it appears in the image.

However, many types of data possess a fixed, non-grid-like structure where spatial position is critical. In functional MRI (fMRI) scans, for example, anatomical regions are fixed; the visual cortex is always at the back of the brain, and the motor cortex is in the middle. A standard CNN struggles with this data because it applies the same filters across the entire "image," ignoring these fixed anatomical realities. Conversely, a Fully Connected Network (FNN) ignores spatial relationships entirely, resulting in a massive parameter count and a high risk of overfitting.

To bridge this gap, researchers introduce the Fixed Grouping Layer (FGL), a novel neural network component designed to bake "structured smoothness" directly into a model's architecture. By restricting connections between layers based on predefined spatial or functional groups, FGL outperforms standard convolutional and fully connected networks on complex spatial data.

The Fixed Grouping Layer (FGL)

The core innovation of this paper is the FGL. Instead of connecting every input to every output (like an FNN) or sliding a uniform filter across the data (like a CNN), FGL uses a predefined "grouping matrix" ($A$) derived from domain knowledge. For neuroimaging, this matrix is based on a brain parcellation map created via clustering techniques.

Hierarchical brain parcellations used to define grouping matrices.

Mathematically, the layer is defined as: $$z = A((xv) \odot u) + b$$

  • $x$: The input data.
  • $v$: A linear transformation projecting input features into a new space.
  • $u$: A rescaling matrix applying a Hadamard product (element-wise multiplication) to weight the importance of specific inputs.
  • $A$: The binary grouping matrix that aggregates these processed signals into specific output groups.

By stacking these layers, the network learns hierarchically. Voxels are grouped into small clusters, which are subsequently grouped into larger functional regions, eventually leading to a final classification.

Datasets and Evaluation

The authors evaluated the FGL architecture using both synthetic and real-world datasets to test its robustness and real-world applicability.

Simulated data generated using Voronoi diagrams.
  • Simulated Dataset: Images were generated using Voronoi diagrams to create spatially connected but irregularly shaped regions, testing the model's ability to handle non-grid spatial structures.
  • Real fMRI Datasets: The model was tested on five distinct neuroimaging datasets for "brain decoding" (predicting a subject's task based on brain activity). These included smaller datasets (Archi, Brainomics, Cam-CAN, and LA5c) and the massive Human Connectome Project (HCP) dataset, which features over 18,000 images from 787 subjects.

Key Insights and Results

FGL demonstrates that for structured spatial data, hard-coding the "where" into the architecture yields significant benefits:

  • State-of-the-Art Performance: On the large HCP dataset, a 3-layer FGL model achieved 93.36% accuracy, significantly outperforming standard CNNs (91.38%) and FNNs (92.16%).
  • Sample Efficiency: Because FGL does not have to learn the anatomy of the brain from scratch, it performs exceptionally well when training data is scarce. On the smaller Archi dataset, FGL reached 87.07% accuracy, a nearly 3% improvement over the best-performing CNN baseline.
  • Robustness to Noise: In simulated tests, FGL maintained high accuracy even when labels were noisy, whereas CNNs and Logistic Regression models experienced sharper performance drops.
  • Depth vs. Width: Increasing the intermediate vector length (features extracted per group) improves performance up to a point. However, adding more FGL layers (depth) provided a statistically significant boost on the large HCP dataset.

Limitations and Future Work

While highly effective, FGL's success is heavily dependent on pre-processing. It requires a high-quality grouping matrix ($A$); if the initial clustering or brain parcellation is poor, the layer's performance will suffer. Additionally, unlike CNNs which share weights across an entire image, FGL's current formulation does not easily allow for parameter sharing between different groups, though constant group sizes could mitigate this in the future.

Finally, while this study focuses on neuroimaging, the inductive bias of structured smoothness is broadly applicable. Future research could apply FGL to other domains with fixed structures, such as financial markets (grouping stocks by sector) or socio-economic data (grouping by geography).

Summary updated

Abstract

We propose the Fixed Grouping Layer (FGL); a novel feedforward layer designed to incorporate the inductive bias of structured smoothness into a deep learning model. FGL achieves this goal by connecting nodes across layers based on spatial similarity. The use of structured smoothness, as implemented by FGL, is motivated by applications to structured spatial data, which is, in turn, motivated by domain knowledge. The proposed model architecture outperforms conventional neural network architectures across a variety of simulated and real datasets with structured smoothness.
BibTeX
@inproceedings{habeeb2020towards,
  author = {Haroun Habeeb and Oluwasanmi Koyejo},
  title = {{Towards a Deep Network Architecture for Structured Smoothness}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=vajcbIwmic_},
}
Figure