INSIDE: Steering Spatial Attention with Non-imaging Information in CNNs

Grzegorz Jacenków, Alison Q. O'Neil, Brian Mohr, Sotirios A. Tsaftaris

Abstract

Standard Convolutional Neural Networks (CNNs) often overlook valuable non-imaging metadata, such as patient demographics or acquisition parameters, which human experts use to contextualize images. We present INSIDE (INstance modulation with SpatIal DEpendency), a novel conditioning layer that "steers" spatial attention using such metadata. By integrating a parameter-efficient Gaussian attention mechanism with feature-wise linear modulation (FiLM), INSIDE allows networks to learn localized regions of interest. Our experiments on the CLEVR-Seg and ACDC datasets show that INSIDE significantly improves segmentation performance, especially in data-scarce scenarios where metadata acts as a vital inductive bias.

Introduction

In medical imaging, scans are rarely interpreted in isolation. Clinical context—such as the laterality of a symptom or the specific slice position in a 3D volume—provides essential cues for diagnosis. While existing methods like FiLM allow metadata to scale or shift feature maps, they are spatially "blind," treating the entire image uniformly. INSIDE bridges this gap by providing conditioning layers with a spatial coordinate system, enabling the model to translate categorical metadata into precise spatial focus.

Methodology: The INSIDE Layer

The core of the INSIDE architecture is the application of a spatial attention mask to feature maps prior to standard modulation.

Figure 1: The INSIDE architecture integrates an auxiliary network that predicts Gaussian attention parameters and FiLM parameters from a conditioning vector.
  1. Gaussian Attention: To maintain parameter efficiency and avoid overfitting, INSIDE uses a 2D Gaussian function as an attention mechanism. An auxiliary hypernetwork (a small MLP) predicts the center ($\mu$) and spread ($\sigma$) for both axes for each feature map.
  2. Modulation: The resulting attention mask is multiplied with the feature maps, followed by FiLM-style scaling ($\gamma$) and shifting ($\beta$).
  3. Training: The system is fully differentiable. An $L_2$ regularization penalty on the Gaussian's standard deviation ($\sigma$) encourages the model to learn "tight," localized attention rather than defaulting to global coverage.

Experimental Evaluation

The method was evaluated on two benchmarks:

  • CLEVR-Seg: A synthetic dataset where the model segments objects based on commands like "bottom-right quadrant" or "small green cube."
  • ACDC (Automated Cardiac Diagnosis Challenge): A medical dataset of cardiac cine-MRI. Metadata includes slice position and cardiac phase.

Key Results

INSIDE consistently outperforms baseline and spatially-blind conditioning methods. On CLEVR-Seg's quadrant segmentation task, INSIDE achieved a Dice score of 0.857, nearly doubling the performance of FiLM (0.487).

Figure 4: Qualitative results on the ACDC dataset showing how the INSIDE layer produces localized attention masks that adapt to anatomy.

In the medical domain, the Gaussian attention acts as an effective "shape prior" for the heart. Notably, INSIDE demonstrates high robustness in low-data regimes. When trained on only 6% of the ACDC dataset, INSIDE maintained a Dice score of ~62%, a 16% relative improvement over the baseline (53.6%), suggesting that metadata can effectively compensate for a lack of labeled images.

Discussion and Future Work

INSIDE provides a direct "hook" for external spatial context in CNNs. While the current use of a single Gaussian per channel is efficient, it may be limited for complex, non-convex structures. Future research could explore more flexible differentiable distributions or extend the mechanism to 3D volumes by adding a third Gaussian component for the z-axis. Despite these limitations, the parameter efficiency and performance gains in low-data settings make INSIDE a promising tool for clinical applications where metadata is abundant but labeled data is scarce.

Summary updated

Abstract

We consider the problem of integrating non-imaging information into segmentation networks to improve performance. Conditioning layers such as FiLM provide the means to selectively amplify or suppress the contribution of different feature maps in a linear fashion. However, spatial dependency is difficult to learn within a convolutional paradigm. In this paper, we propose a mechanism to allow for spatial localisation conditioned on non-imaging information, using a feature-wise attention mechanism comprising a differentiable parametrised function (e.g. Gaussian), prior to applying the feature-wise modulation. We name our method INstance modulation with SpatIal DEpendency (INSIDE). The conditioning information might comprise any factors that relate to spatial or spatio-temporal information such as lesion location, size, and cardiac cycle phase. Our method can be trained end-to-end and does not require additional supervision. We evaluate the method on two datasets: a new CLEVR-Seg dataset where we segment objects based on location, and the ACDC dataset conditioned on cardiac phase and slice location within the volume. Code and the CLEVR-Seg dataset are available at https://github.com/jacenkow/inside.
BibTeX
@inproceedings{jacenkow2020inside,
  author = {Grzegorz Jacenków and Alison Q. O'Neil and Brian Mohr and Sotirios A. Tsaftaris},
  title = {{INSIDE: Steering Spatial Attention with Non-imaging Information in CNNs}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  url = {https://openreview.net/forum?id=JRKeCqs8tX},
}
Figure