Geo-SIC: Learning Deformable Geometric Shapes in Deep Image Classifiers

Jian Wang, Miaomiao ZhangNeurIPS 2022

Presented at NeurIPS 2022, Geo-SIC introduces a framework that bridges the gap between classical geometric shape analysis and modern deep learning classifiers. By teaching a neural network to "understand" how shapes deform, the authors achieve better accuracy, higher robustness to noise, and—crucially—models that are much easier for humans to interpret.

The Challenge with Standard CNNs

Deep learning has revolutionized image classification, but standard Convolutional Neural Networks (CNNs) often treat geometric shape as an implicit byproduct of pixel intensities. While shapes are highly robust to noise and texture variations, most models either ignore the intricate interior structures of objects or require expensive "pre-extracted" shape data, such as landmarks or segmentations.

Furthermore, medical imaging—such as 3D brain MRIs—relies heavily on identifying subtle geometric changes, like the shrinkage of specific brain regions in Alzheimer’s disease. Standard "black-box" classifiers often fail to provide the interpretability clinicians need to trust a diagnosis. Geo-SIC addresses this by explicitly modeling these geometric deformations as part of the learning process.

The Geo-SIC Architecture

Geo-SIC is the first deep learning model to jointly learn deformable geometric shapes and image classification in a single, end-to-end framework. The core of Geo-SIC is a dual-module architecture:

Geo-SIC Architecture
  1. Geometric Shape Learning (Atlas Building): This sub-network performs unsupervised learning to find a "template" (atlas) for each class and the "velocity fields" (deformations) required to warp that template to match an input image.
  2. Boosted Classification Network: This module extracts features from both the original image and the learned latent shape space. These features are fused to make the final class prediction.

Key Technical Innovations

  • Diffeomorphic Transformations: The model uses "diffeomorphisms"—mappings that are smooth and invertible. This ensures that the topology of objects (like a brain or a heart) remains intact during deformation, preventing unrealistic tearing or folding in the model's internal representation.
  • Low-Dimensional Fourier Space: To solve the massive computational hurdle of 3D deformations, the authors parameterize the transformations in a bandlimited Fourier space. This significantly speeds up training and reduces memory consumption compared to traditional high-dimensional methods.
  • Joint Optimization: Unlike "two-step" approaches where shape is extracted first, Geo-SIC uses an alternating optimization scheme to learn the most relevant shape features specifically for the classification task.

Datasets and Evaluation

The authors evaluated Geo-SIC on two distinct types of data:

  • 2D Synthetic Data: 50,000 images from the Google Quickdraw dataset, focusing on five classes: circle, cloud, envelope, square, and triangle.
  • 3D Medical Data: 373 public T1-weighted brain MRI scans from the OASIS dataset, categorized as either "demented" (Alzheimer’s) or "non-demented."

Key Insights and Results

Interpretability and Visual Quality

By using Grad-CAM to visualize what the network attends to, the authors found that Geo-SIC naturally focuses on the edges and geometric boundaries of objects. In brain MRIs, it correctly identified the ventricles and white/grey matter boundaries—regions known to be critical for Alzheimer’s diagnosis.

Grad-CAM and Atlas Visualizations

Additionally, the "atlases" (mean shapes) generated by Geo-SIC are visually sharper and more anatomically accurate than those produced by competing unsupervised methods like Lagomorph.

Performance and Robustness

  • Superior Accuracy: On 2D data, Geo-SIC (using a ResNet backbone) achieved 93.5% accuracy, compared to 87.5% for a standard ResNet18.
  • Clinical Potential: On the 3D brain MRI task, Geo-SIC achieved an AUC of 0.930, significantly outperforming VGG19 (0.871) and ResNet18 (0.887).
  • Robustness to Noise: The model demonstrates that geometric features are far more stable than pixel intensities. Under heavy adversarial attack ($\epsilon = 0.5$), Geo-SIC maintained roughly 10% higher accuracy than baseline models because it "looked" at the underlying structure rather than just the noisy pixels.

Limitations and Future Work

While powerful, the current version of Geo-SIC uses a single template per class. This might struggle with "multimodal" classes—for example, a "chair" class that includes both stools and armchairs. The authors suggest that future work could involve modeling multiple templates per class to capture this diversity. Additionally, they aim to adapt the model to handle images with missing data (occlusions) and apply it to dynamic processes like tracking tumor growth over time.

Summary updated

BibTeX
@inproceedings{wang2022geo,
  author = {Jian Wang and Miaomiao Zhang},
  title = {{Geo-SIC: Learning Deformable Geometric Shapes in Deep Image Classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure