TL;DR
This paper introduces ALTBI, a novel unsupervised outlier detection (UOD) method that significantly improves performance by optimizing the "inlier-memorization" (IM) effect. ALTBI achieves state-of-the-art results by gradually increasing mini-batch sizes and using adaptive loss truncation during training, making outlier identification more robust and efficient.
What Problem Does This Paper Address?
Outlier detection (OD) is a crucial task in many fields, aiming to identify unusual data points that deviate from normal patterns. Unsupervised Outlier Detection (UOD) is particularly challenging because it operates without any prior knowledge or labels about which data points are outliers. While deep generative models (DGMs) have shown promise in UOD, a recent observation called the "inlier-memorization" (IM) effect suggests that DGMs tend to learn inliers before outliers during early training stages. This means that under-fitted DGMs can be used to identify outliers based on their per-sample loss values. However, existing methods leveraging this effect, like ODIM, have limitations in finding the optimal training point and can be computationally expensive due to ensemble techniques. This paper aims to build upon the IM effect to create a more principled, efficient, and effective UOD method.
Methods
The core of ALTBI lies in two key techniques designed to maximize the IM effect:
- Mini-batch Increment: Instead of using a fixed mini-batch size throughout training, ALTBI gradually increases the mini-batch size. This helps to expose the model to more data over time, allowing for a more stable learning process and a clearer distinction between inliers and outliers.
- Adaptive Loss Truncation: ALTBI employs a truncated loss function. This means that at each training step, only a portion of the samples with the lowest per-sample loss values are used to compute the loss. An adaptive threshold, determined by the percentile of per-sample losses, is used to decide which samples to include. This effectively filters out potential outliers from the loss calculation during early training, further enhancing the IM effect.
Additionally, ALTBI incorporates an ensemble strategy within a single model. Instead of training multiple separate models, it averages the per-sample loss values from a range of training updates (T1 to T2). This approach leverages the IM effect across different stages of training without the computational overhead of training multiple DGMs.
Datasets
The paper evaluates ALTBI on a comprehensive set of 57 outlier detection benchmark datasets from ADBench. These datasets cover various domains:
- Tabular data: 46 datasets from healthcare, finance, and astronautics.
- Natural Language Processing (NLP): 5 datasets (20news, Agnews, Amazon, IMDB, Yelp), using BERT or RoBERTa embeddings.
- Image data: 6 datasets (CIFAR10, MNIST-C, MVTec-AD, SVHN, MNIST, FMNIST), using ViT embeddings.
The authors also mention using the PageBlocks dataset for a specific experiment on the relationship between outlier ratio and the IM effect.
Key Insights and Ideas
- IM Effect Clarity: The paper's key insight is that the inlier-memorization (IM) effect is more pronounced when the training data contains fewer outliers. This observation is crucial for understanding why filtering out outliers during training can boost performance.
- Filtering for Clarity: The core idea is that by actively filtering out potential outliers from the loss function during early training stages, we can amplify the IM effect. This is achieved through the mini-batch increment and adaptive loss truncation.
- Efficiency through Ensemble: The ensemble strategy within a single model is a clever way to gain the benefits of ensembling (stability and performance) without the significant computational cost of training multiple independent models.
- Robustness with DP: ALTBI shows promising robustness when combined with Differential Privacy (DP) techniques, suggesting its potential for privacy-preserving anomaly detection.
Key Results
- ALTBI achieves state-of-the-art performance in outlier detection across the 57 evaluated datasets, outperforming existing methods like ODIM.
- The method demonstrates high computational efficiency, requiring only a simple and under-fitted likelihood-based DGM (like VAE or normalizing flows) for a limited number of updates.
- Experiments show that the mini-batch increment and adaptive threshold significantly improve the clarity of the IM effect, leading to superior outlier detection AUC values.
- The ensemble approach within a single model provides stability to the trained model, reducing fluctuations in performance.
- ALTBI shows robust performance when combined with DP-SGD, indicating a synergistic effect.
Limitations and Future Work
- Heuristic Nature of ODIM's Optimal Update: The paper notes that ODIM's heuristic approach to finding the optimal update point (based on bi-modality) can sometimes fail, leading to suboptimal performance. ALTBI's ensemble approach addresses this by not relying on a single optimal update.
- Future Work on Labeled Outliers: The authors suggest that extending their method to scenarios where a few outliers with known information are accessible (semi-supervised OD) would be an interesting future direction. They also mention the potential for applying ALTBI to cases where labeled outliers might be wrongly annotated.
Summary updated