Permutohedral Attention Module for Efficient Non-Local Neural Networks

Samuel Joutard, Reuben Dorent, Amanda Isaac, Sébastien Ourselin, Tom Vercauteren, Marc Modat

Abstract

Medical image processing tasks, such as segmentation, often require capturing non-local information. Because organs, bones, and tissues share common characteristics like intensity, shape, and texture, contextual information is critical for correct labeling. While Convolutional Neural Networks (CNNs) are the standard for these tasks, their context is limited by the receptive field, which is constrained by memory and computational requirements. We propose the Permutohedral Attention Module (PAM), a novel attention mechanism designed to efficiently capture non-local characteristics in 3D medical images. PAM is both memory and computationally efficient, scaling linearly with the number of voxels. We demonstrate its effectiveness on the challenging task of vertebrae segmentation and labeling, where global context is essential for distinguishing between visually similar bones.

Introduction

In medical imaging, context is everything. A single vertebra in a CT scan looks remarkably like its neighbors; to label them correctly, a model must understand where one bone sits relative to the entire spine. Standard CNNs have a limited "receptive field," looking only at local neighborhoods. While stacking layers or using dilated convolutions can expand this view, these methods often become computationally expensive or lose fine-grained detail.

Non-local or self-attention mechanisms are the gold standard for capturing global context, but they typically require calculating interactions between every pair of voxels. For a 3D volume, this leads to a quadratic complexity ($O(N^2)$) in memory and computation, making them impractical for high-resolution medical data. The Permutohedral Attention Module (PAM) addresses this by providing a linear-time ($O(N)$) approximation of the attention mechanism.

Methodology: The Permutohedral Attention Module

The PAM bridges the gap between local and global context by using an efficient approximation algorithm based on the Permutohedral Lattice. The module transforms the attention operation into a high-dimensional filtering problem, which can be solved in four distinct, parallelizable steps:

  1. Position Embedding: Features are mapped into a high-dimensional space to define their relative positions.
  2. Splat: Features are "distributed" or projected onto the vertices of a sparse permutohedral lattice.
  3. Blur: A Gaussian blur is applied along the lattice dimensions. This step effectively performs the weighted averaging characteristic of self-attention.
  4. Slice: The filtered information is interpolated back from the lattice vertices to the original voxel positions.
The four-step process of the Permutohedral Attention Module: Position Embedding, Splat, Blur, and Slice.

By deriving the gradients for this entire process, the authors ensure the module is end-to-end trainable. This allows the network to learn exactly which features are most important for determining non-local relationships. Furthermore, the authors implemented a "channel splitting" technique, applying PAM independently to groups of channels to further reduce the memory footprint.

Experimental Setup and Results

The module was evaluated using the CSI 2014 workshop challenge data, consisting of 20 CT images of the human spine. The task involved the simultaneous segmentation and labeling of Cervical, Thoracic, and Lumbar vertebrae.

Quantitative results demonstrated that adding PAM provided a significant performance boost across different architectures:

Model Mean Dice Score
Baseline FCN 28%
FCN + PAM 54%
Standard 3D U-Net 72%
U-PAM-Net (U-Net + PAM) 81%

The improvement was most notable for cervical vertebrae, which are small and often poorly handled by standard U-Nets due to aggressive down-sampling. U-PAM-Net improved the Cervical Dice score from 25% to 55%. The results suggest that efficient global communication can sometimes replace the need for massive architectural depth; a shallow 6-layer network with PAM performed nearly as well as a much deeper standard 3D U-Net.

Conclusion

The Permutohedral Attention Module offers a scalable and efficient way to integrate global context into 3D medical imaging pipelines. By reducing the complexity of self-attention from quadratic to linear, PAM enables the processing of high-resolution volumes that were previously inaccessible to non-local networks. Future work will explore the use of asymmetric attention matrices to further increase the model's representational power and optimize GPU implementations for even larger 3D datasets.

Summary updated

Abstract

Medical image processing tasks such as segmentation often require capturing non-local information. As organs, bones, and tissues share common characteristics such as intensity, shape, and texture, the contextual information plays a critical role in correctly labeling them. Segmentation and labeling is now typically done with convolutional neural networks (CNNs) but the context of the CNN is limited by the receptive field which itself is limited by memory requirements and other properties. In this paper, we propose a new attention module, that we call Permutohedral Attention Module (PAM), to efficiently capture non-local characteristics of the image. The proposed method is both memory and computationally efficient. We provide a GPU implementation of this module suitable for 3D medical imaging problems. We demonstrate the efficiency and scalability of our module with the challenging task of vertebrae segmentation and labeling where context plays a crucial role because of the very similar appearance of different vertebrae.
BibTeX
@inproceedings{joutard2019permutohedral,
  author = {Samuel Joutard and Reuben Dorent and Amanda Isaac and Sébastien Ourselin and Tom Vercauteren and Marc Modat},
  title = {{Permutohedral Attention Module for Efficient Non-Local Neural Networks}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=AbO1WQEBGsx},
}
Figure