SATr: Slice Attention with Transformer for Universal Lesion Detection

Han Li, Long Chen, Hu Han, S. Kevin Zhou

Abstract

Universal Lesion Detection (ULD) in CT scans is a challenging task due to the immense variety of lesion types, sizes, and anatomical locations. While Convolutional Neural Networks (CNNs) are effective at local feature extraction, they often struggle to capture the global, 3D spatial context that radiologists use when scrolling through adjacent slices. We present the Slice Attention Transformer (SATr), a "plug-and-play" module designed to integrate global modeling into existing CNN backbones. By utilizing a novel slice-filtering mechanism, SATr forces the model to learn dependencies between a central "key" slice and its neighbors. Evaluated on the DeepLesion dataset, SATr consistently improves the performance of state-of-the-art detection frameworks, achieving new benchmark results and demonstrating superior data efficiency.

Introduction

Universal Lesion Detection (ULD) involves the localization of a wide variety of abnormalities—ranging from lung nodules to liver masses—within Computed Tomography (CT) scans. Unlike specialized detection tasks, ULD must account for significant variations in lesion morphology. In clinical practice, radiologists interpret these lesions by scrolling through multiple axial slices to understand their 3D structure.

However, most AI models rely on 2D CNNs which, despite their efficacy in texture recognition, possess a "local" inductive bias that limits their ability to model long-distance spatial relationships across slices. While 3D CNNs offer a potential solution, they are often hampered by a lack of large-scale, labeled 3D medical datasets for effective pre-training. SATr addresses this by providing a hybrid architecture that combines the strengths of CNNs and Transformers.

Methodology

The Slice Attention Transformer (SATr) block is designed to be inserted into existing CNN backbones (such as DenseNet) to create a hybrid architecture. The SATr block operates in two primary stages:

  1. Patch Embedding Extraction: The input CT slices are processed through a combination of 2D and 3D convolutions followed by pooling. This step transforms the raw image data into "patches" that a Transformer can process.
  2. Slice Feature Attention: This stage employs a modified version of Multi-Head Self-Attention (MSA). To optimize it for medical imaging, the authors introduced two key modifications:
  3. Key-Slice Value Enhancement: The features of the central "key slice" (where the lesion is primarily localized) are explicitly added to the "Value" vector. This ensures the model retains high-fidelity information from the primary slice while attending to context.
  4. Q/K Filtering: The key slice's features are removed from the "Query" and "Key" vectors. This forces the model to learn dependencies between the key slice and its neighbors, preventing the network from becoming "lazy" and only looking at the central slice.

Experimental Setup

The researchers evaluated their method using DeepLesion, currently the largest publicly available multi-category lesion dataset. The dataset includes 32,735 lesions across 32,120 axial CT slices from over 4,400 patients. It features a wide variety of lesions (lung, liver, bone, kidney, etc.) with diameters ranging from 0.21mm to 342.5mm.

Results and Discussion

The authors tested SATr by plugging it into five different state-of-the-art (SOTA) ULD methods. In every case, performance improved:

  • Sensitivity Boost: On the DeepLesion test set, adding SATr improved the average sensitivity of baseline models by 1.38% to 4.62%.
  • New SOTA: The combination of the "A3D" model plus the SATr block achieved a new state-of-the-art result for the DeepLesion benchmark.
  • Data Efficiency: SATr was particularly effective when training data was scarce. When using only 25% of the training data, the SATr-enhanced models showed even more dramatic improvements over the baselines, suggesting it helps the model learn more robust features.

By forcing the model to use 3D context through Q/K filtering, the authors found that the Transformer actually learns the spatial relationships rather than overfitting to the central slice. This hybrid approach allows the CNN to handle fine-grained textures while the Transformer manages the "big picture" spatial relationships.

Limitations and Future Work

While SATr provides significant gains, there are areas for further development:

  • Computational Overhead: Transformers are generally more computationally expensive than pure CNNs, and a detailed breakdown of inference time or memory usage was not provided.
  • Slice Range: The study primarily focused on 3, 7, or 9-slice inputs. Future work could explore how SATr scales to larger "z-axis" contexts to better mimic clinical workflows.
  • Automation: While "plug-and-play," the block still requires manual placement within a backbone. Automating the placement of such hybrid blocks via Neural Architecture Search (NAS) could be a promising next step.

Summary updated

Abstract

Universal Lesion Detection (ULD) in computed tomography plays an essential role in computer-aided diagnosis. Promising ULD results have been reported by multi-slice-input detection approaches which model 3D context from multiple adjacent CT slices, but such methods still experience difficulty in obtaining a global representation among different slices and within each individual slice since they only use convolution-based fusion operations. In this paper, we propose a novel Slice Attention Transformer (SATr) block which can be easily plugged into convolution-based ULD backbones to form hybrid network structures. Such newly formed hybrid backbones can better model long-distance feature dependency via the cascaded self-attention modules in the Transformer block while still holding a strong power of modeling local features with the convolutional operations in the original backbone. Experiments with five state-of-the-art methods show that the proposed SATr block can provide an almost free boost to lesion detection accuracy without extra hyperparameters or unique network designs. Code: https://github.com/MIRACLE-Center/A3D_SATr .
BibTeX
@inproceedings{li2022satr,
  author = {Han Li and Long Chen and Hu Han and S. Kevin Zhou},
  title = {{SATr: Slice Attention with Transformer for Universal Lesion Detection}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=XxpAjeMrCR},
}
Figure