Efficient Pairwise Neuroimage Analysis Using the Soft Jaccard Index and 3D Keypoint Sets

Laurent Chauvin, Kuldeep Kumar, Christian Desrosiers, William M. Wells Iii, Matthew Toews

TL;DR

Researchers have developed a new way to compare brain scans by treating them as sets of 3D "keypoints" and measuring their similarity using a "Soft Jaccard Index." This method is highly efficient, scaling to massive datasets, and is sensitive enough to identify identical twins with near-perfect accuracy and predict family relationships based solely on brain structure.

What Problem Does This Paper Address?

Analyzing large-scale neuroimaging datasets (like the Human Connectome Project) usually requires intense preprocessing, such as segmenting specific brain structures or warping every brain to a standard template. These steps are computationally expensive and can "smooth away" the unique, fine-grained anatomical patterns that distinguish individuals or families.

Existing methods often use "hard" matching—either two points in a brain are the same, or they aren't. However, medical images are noisy, and biological structures vary slightly even between twins. This paper addresses the need for a pairwise distance measure that is:

  1. Robust: Able to handle the uncertainty and "soft" nature of biological similarity.
  2. Efficient: Capable of comparing thousands of images without the quadratic $O(N^2)$ time complexity that usually plagues pairwise analysis.
  3. Discriminative: Sensitive enough to link a person's brain structure (phenotype) to their genetic makeup (genotype).

Methods

The authors propose a framework based on 3D SIFT-Rank keypoints. Think of these as "landmarks" automatically detected at salient locations in the brain (like specific folds or corners of the ventricles).

The core innovations are:

  • Soft Jaccard Index ($J_{SSE}$): The standard Jaccard index measures "Intersection over Union" for sets. The authors generalized this into a "soft" version. Instead of a binary "yes/no" for matching keypoints, they use a kernel function to assign a similarity score between 0 and 1.
  • New Geometry Kernel: While previous work focused on the appearance of these landmarks, this paper introduces a kernel that accounts for geometry. It penalizes landmarks that are too far apart or have different scales, but it does so adaptively—larger landmarks are allowed more "wiggle room" in their location than tiny ones.
  • Efficient Indexing: By using KD-trees for nearest-neighbor lookups, the system avoids comparing every keypoint in Brain A to every keypoint in Brain B. This brings the complexity down to $O(N \log N)$, making it practical for datasets with thousands of subjects.

Datasets

The study utilized the Human Connectome Project (HCP) Q4 release, which includes:

  • 1,010 unique subjects (ages 22–36).
  • 434 unique families, including monozygotic (identical) twins, dizygotic (fraternal) twins, full siblings, and half-siblings.
  • T1-weighted MRI volumes preprocessed with standard skull-stripping and alignment.

Key Insights and Ideas

  • The Brain as a Genetic Fingerprint: The results reinforce the idea that brain morphology is tightly coupled with genetics. The "closer" two people are genetically (MZ twins > siblings > unrelated), the lower their Soft Jaccard distance.
  • Soft vs. Hard Equivalence: "Hard" matching (HSE) often fails because it can't account for the probabilistic nature of where a landmark is detected. "Soft" matching (SSE) acts as a regularizer, significantly improving the ability to distinguish fraternal twins and siblings.
  • Geometry is the Secret Sauce: Adding the geometry kernel (location and scale) to the appearance-based matching provided a major boost in accuracy. It helps filter out "geometrically inconsistent" matches—points that look similar but are in completely different parts of the brain.
  • Data Curation Potential: The method was so sensitive it identified three subjects who were labeled as "unrelated" in the dataset but were actually siblings (likely due to inconclusive genotyping). This suggests the tool is excellent for "cleaning" large datasets.

Key Results

  • Twin Identification: Monozygotic (identical) twin identification was near 100%.
  • Sex Prediction: By comparing an individual brain to a "superset" of male and female brains, the model predicted sex with an AUC of 0.97.
  • Family Prediction: The Soft Jaccard Index consistently outperformed the standard Jaccard index across all sibling categories.
  • Speed: Keypoint extraction takes only ~3 seconds per image, and querying a subject against the entire database takes less than a second.

Limitations and Future Work

  • Spatial Alignment: While the appearance kernel is invariant to rotation and scale, the geometry kernel requires images to be roughly aligned to a common template (atlas). The authors note that while the method is robust to different alignment techniques, some level of normalization is currently necessary.
  • Pairwise vs. Group: The current work focuses on comparing pairs of subjects. The authors suggest that future work could use graph theory to model entire family "cliques" or clusters within a population.
  • Domain Specificity: The keypoints used are generic. The authors mention that training domain-specific keypoint models (e.g., specifically for the cortex or subcortical structures) could further improve performance for specific diseases or aging studies.

Summary updated

Abstract

We propose a novel pairwise distance measure between image keypoint sets, for the purpose of large-scale medical image indexing. Our measure generalizes the Jaccard index to account for soft set equivalence (SSE) between keypoint elements, via an adaptive kernel framework modeling uncertainty in keypoint appearance and geometry. A new kernel is proposed to quantify the variability of keypoint geometry in location and scale. Our distance measure may be estimated between <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${O}\,{(}{N}^{{\,{2}}}{)}$ </tex-math></inline-formula> image pairs in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${O}\,{(}{N}\,\text {log}{N}\,{)}$ </tex-math></inline-formula> operations via keypoint indexing. Experiments report the first results for the task of predicting family relationships from medical images, using 1010 T1-weighted MRI brain volumes of 434 families including monozygotic and dizygotic twins, siblings and half-siblings sharing 100%-25% of their polymorphic genes. Soft set equivalence and the keypoint geometry kernel improve upon standard hard set equivalence (HSE) and appearance kernels alone in predicting family relationships. Monozygotic twin identification is near 100%, and three subjects with uncertain genotyping are automatically paired with their self-reported families, the first reported practical application of image-based family identification. Our distance measure can also be used to predict group categories, sex is predicted with an AUC = 0.97. Software is provided for efficient fine-grained curation of large, generic image datasets.
BibTeX
@article{chauvin2022efficient,
  author = {Laurent Chauvin and Kuldeep Kumar and Christian Desrosiers and William M. Wells III and Matthew Toews},
  title = {{Efficient Pairwise Neuroimage Analysis Using the Soft Jaccard Index and 3D Keypoint Sets}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2022},
  url = {https://openreview.net/forum?id=ugFM1tsEa9W},
}
Figure