OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data

Christoph Reich, Tim Prangemeier, Özdemir Çetin, Heinz Koeppl

Introduction

High-resolution 3D medical imaging is essential for accurate diagnosis and treatment planning. However, the standard approach for segmentation—3D Convolutional Neural Networks (CNNs) like the 3D U-Net—faces a significant "memory wall." Because memory consumption grows cubically ($O(N^3)$) with resolution, processing native $512^3$ volumes often requires extreme downsampling or small patches, both of which sacrifice anatomical detail.

OSS-Net (Occupancy Networks for Semantic Segmentation) addresses this by combining the feature-extraction power of 3D CNNs with the memory efficiency of implicit neural representations, enabling high-resolution processing on consumer-grade hardware.

The OSS-Net Architecture

OSS-Net bridges the gap between voxel-based grids and continuous occupancy functions. The architecture is composed of three primary components designed to capture both global context and local detail:

  1. 3D CNN Encoder: This component processes a downscaled version of the input volume to generate a global latent vector, providing the model with a broad understanding of the anatomy.
  2. Patch Encoder: To recover fine-grained details, the model samples local 3D patches around specific coordinates. These patches are encoded into local latent vectors.
  3. Occupancy Decoder: A fully connected network (comprising 5 residual blocks) takes the concatenated global vector, local patch features, and 3D coordinates to predict the occupancy probability for any given point.
The OSS-Net architecture combines a 3D CNN encoder for global context with a patch encoder for local detail, feeding into a residual occupancy decoder.

By utilizing local observations, OSS-Net can represent complex, jagged structures like tumors more accurately than traditional Occupancy Networks, which often produce overly smoothed results.

Key Results and Performance

The model was rigorously tested on the BraTS 2020 (brain tumor) and LiTS (liver tumor) datasets. A major innovation in OSS-Net is its expedited inference strategy. Instead of evaluating every point in a volume, it uses the low-resolution "rough draft" from the encoder to skip empty space and focus only on boundaries, making it 2x faster than previous implicit models.

Qualitative results showing 2D slices and 3D reconstructions of brain tumor segmentations. OSS-Net captures complex anatomical structures with high fidelity.

Performance Highlights:

  • Accuracy: On BraTS, OSS-Net achieved a Dice score of 0.8842, matching the performance of the memory-intensive 3D Residual U-Net (0.8827).
  • Memory Efficiency: The model is 5x more memory-efficient during training and 10x more efficient during inference.
  • Scalability: While a standard 3D CNN would require over 110 GB of VRAM for a $512^3$ volume, OSS-Net operates comfortably within a 16 GB GPU envelope.

Insights and Future Directions

The success of OSS-Net suggests that "prior knowledge" is key to making implicit models practical. By initializing the occupancy search with a low-res prediction, the computational overhead is drastically reduced. Furthermore, the authors found that sampling strategies are task-dependent: scattered brain tumors benefit from uniform sampling, while large organs like the liver benefit from focusing on boundaries.

Future work will likely explore adaptive sampling, where the network dynamically decides which areas require higher density sampling based on its own prediction confidence, further optimizing the balance between speed and precision.

Summary updated

BibTeX
@inproceedings{reich2021oss,
  author = {Christoph Reich and Tim Prangemeier and Özdemir Çetin and Heinz Koeppl},
  title = {{OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data}},
  booktitle = {BMVC},
  year = {2021},
}
Figure