Topological Structure Learning for Weakly-Supervised Out-of-Distribution Detection

Rundong He, Rongxue Li, Zhongyi Han, Xihong Yang, Yilong Yin

TL;DR

Detecting "Out-of-Distribution" (OOD) data is vital for safe AI, but most models rely on massive amounts of labeled data that are expensive to collect. This paper introduces Topological Structure Learning (TSL), a method designed for a "weakly-supervised" setting where labeled data is scarce but unlabeled data is plentiful. By mining reciprocal neighbor connections and reconstructing a new "topological space," TSL significantly outperforms current state-of-the-art methods in identifying unknown data.

What Problem Does This Paper Address?

In the "open world," a reliable machine learning model needs the humility to say, "I don't know." This is the essence of Out-of-Distribution (OOD) detection: identifying inputs that don't belong to any of the classes the model was trained on (e.g., a wildlife monitor seeing a new species, or an autonomous car encountering a rare road hazard).

The catch? Most OOD detection methods assume we have a massive, perfectly labeled "In-Distribution" (ID) dataset. In reality, labeling data—especially in specialized fields like medical imaging—requires expensive expert time. The authors tackle a new, more realistic scenario called Weakly-Supervised Out-of-Distribution Detection (WSOOD). In this setting, you have very few labeled ID examples but a large pool of unlabeled data that contains a mix of both ID and OOD instances. The challenge is learning to separate the two when the "prior" model (trained on tiny labeled data) is prone to overfitting and weak feature extraction.

Methods

The authors propose Topological Structure Learning (TSL), which works in three main phases:

  1. Topological Space Representation Learning: Since labels are scarce, TSL starts with SimCLR, a self-supervised contrastive learning method. It uses all available data (labeled and unlabeled) to learn initial feature representations, helping the model understand the "shape" of the data without needing labels.
  2. Topological Connection Mining: This is the "aha" part of the paper. Instead of just looking at nearest neighbors, TSL classifies pairs of data points:
  3. Close Positive Pairs: Reciprocal neighbors (if A is a top neighbor of B, and B is a top neighbor of A). These are highly reliable.
  4. Loose Positive Pairs: One-way neighbors. These are less certain.
  5. Negative Pairs: Points that are far apart based on Mahalanobis distance (a metric that accounts for the distribution's covariance).
  6. Topological Structure Reconstruction: TSL trains a "projector" to map these features into a new space. It first builds a topological skeleton by clustering the few labeled ID examples. Then, it "extends" this skeleton using the mined pairs—pulling positive pairs closer and pushing negative pairs away—to maximize the gap between ID and OOD instances.

Datasets

The researchers evaluated TSL using standard benchmarks, but adapted for the "weakly-supervised" constraint:

  • In-Distribution (ID): CIFAR-10 (only 25 labels per class) and CIFAR-100 (only 4 labels per class).
  • Out-of-Distribution (OOD): TinyImageNet (cropped and resized variants) and LSUN (cropped and resized variants).
  • Unlabeled Mix: A large pool of data containing the remaining training images from the ID sets mixed with OOD images.

Key Insights and Ideas

  • Reciprocity Matters: The move from simple K-Nearest Neighbors to "reciprocal neighbors" is a clever way to handle noise. If two points "choose" each other as neighbors, they are much more likely to belong to the same class than if the relationship is one-sided.
  • Beyond "Everything Else is Negative": Previous methods (like STEP) often treated every other point as a negative instance. TSL is more surgical, using Mahalanobis distance to identify truly distant points as negatives, which prevents the model from accidentally pushing away valid ID points that just happen to be in a different cluster.
  • The Skeleton Metaphor: By treating the tiny labeled set as a "skeleton" and the unlabeled data as the "flesh" that fills out the structure, the model avoids the massive overfitting that usually plagues low-data regimes.

Key Results

TSL consistently outperformed the previous state-of-the-art (STEP) and traditional OOD methods (like ODIN or MAH) across nearly all benchmarks:

  • Accuracy Boost: On the difficult CIFAR-10 & TinyImageNet-resize task, TSL reduced the FPR95 (False Positive Rate at 95% True Positive Rate) by 15.37% compared to STEP.
  • Generalization: TSL showed a remarkable ability to detect "unseen" OOD data—classes that weren't even present in the unlabeled training pool.
  • Robustness: The method proved insensitive to the exact number of neighbors ($K$) chosen, making it easier to deploy without exhaustive hyperparameter tuning.

Limitations and Future Work

While TSL is powerful, the authors note a few areas for growth:

  • Hyperparameter Sensitivity: While robust to $K$, the model's performance drops sharply if the negative mining coefficient ($\beta$) is set too high, suggesting a delicate balance is needed when pushing negative pairs apart.
  • Computational Overhead: Mining reciprocal neighbors and calculating Mahalanobis distances for large unlabeled pools can be computationally intensive.
  • Future Directions: The authors suggest that WSOOD is a burgeoning field and that TSL could be further refined to handle even more complex "real-world" shifts where the ratio of ID to OOD data in the unlabeled pool is unknown or highly skewed.

Summary updated

Abstract

Out-of-distribution (OOD) detection is the key to deploying models safely in the open world. For OOD detection, collecting sufficient in-distribution (ID) labeled data is usually more time-consuming and costly than unlabeled data. When ID labeled data is limited, the previous OOD detection methods are no longer superior due to their high dependence on the amount of ID labeled data. Based on limited ID labeled data and sufficient unlabeled data, we define a new setting called Weakly-Supervised Out-of-Distribution Detection (WSOOD). To solve the new problem, we propose an effective method called Topological Structure Learning (TSL). Firstly, TSL uses a contrastive learning method to build the initial topological structure space for ID and OOD data. Secondly, TSL mines effective topological connections in the initial topological space. Finally, based on limited ID labeled data and mined topological connections, TSL reconstructs the topological structure in a new topological space to increase the separability of ID and OOD instances. Extensive studies on several representative datasets show that TSL remarkably outperforms the state-of-the-art, verifying the validity and robustness of our method in the new setting of WSOOD.
BibTeX
@inproceedings{he2023topological,
  author = {Rundong He and Rongxue Li and Zhongyi Han and Xihong Yang and Yilong Yin},
  title = {{Topological Structure Learning for Weakly-Supervised Out-of-Distribution Detection}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612195},
}
Figure