TL;DR
This paper introduces HPOD, a novel meta-learning framework for optimizing hyperparameters of unsupervised outlier detection (OD) algorithms without requiring ground truth labels. HPOD leverages past performance data from similar tasks to build a predictive model, enabling efficient hyperparameter search in continuous spaces and significantly outperforming existing methods.
What Problem Does This Paper Address?
Unsupervised outlier detection (OD) algorithms are crucial for identifying anomalies in data without explicit labels. However, these algorithms are often highly sensitive to their hyperparameters (HPs), and finding optimal HP settings is challenging because OD tasks typically lack ground truth labels for direct performance evaluation. Practitioners often resort to default HPs, which are rarely optimal, leading to suboptimal detection performance. Existing hyperparameter optimization (HPO) methods, especially those using sequential model-based optimization (SMBO), rely on evaluating performance, which is problematic without labels. This paper addresses the critical need for an effective HPO method for unsupervised OD that can operate without ground truth labels and handle continuous HP spaces.
Methods
HPOD employs a two-phase approach: offline meta-training and online HPO.
- Offline Meta-Training:
- Meta-Database: A large collection of existing OD benchmark datasets with ground truth labels is used.
- Proxy Performance Evaluator (PPE): A regressor,
f(·), is trained to predict the performance of an OD algorithm's HP setting on a given dataset. This PPE takes HP settings, dataset meta-features (characteristics of the dataset), and internal performance measures (IPMs) as input. IPMs are weak, noisy signals derived from outlier scores that can help compare models without ground truth. - Meta-Surrogate Functions (MSFs): For each meta-train dataset, an independent regressor
t(·)is trained to map HP settings directly to performance. These MSFs are later used to improve the online surrogate function. - Online HPO on a New Task:
- SMBO Framework: HPOD adapts the supervised SMBO framework. Instead of using an expensive true objective function (which requires labels), it uses the trained PPE
f(·)to predict HP performance.
- SMBO Framework: HPOD adapts the supervised SMBO framework. Instead of using an expensive true objective function (which requires labels), it uses the trained PPE
- Surrogate Function (
s(·)): A surrogate model (e.g., Gaussian Process) is used to approximate the PPE for fast prediction and uncertainty estimation. - Initialization: The surrogate function is initialized using meta-learning, leveraging knowledge from similar historical tasks to make more accurate initial predictions.
- Acquisition Function: An acquisition function (e.g., Expected Improvement) balances exploration (trying uncertain HPs) and exploitation (focusing on promising HPs) to select the next HP to evaluate using the PPE.
- Surrogate Transfer: To further improve the surrogate's accuracy, HPOD transfers knowledge from the pre-trained MSFs of similar meta-train datasets.
This approach allows HPOD to iteratively identify promising HPs and ultimately select the best one without needing ground truth labels for the target dataset.
Datasets
The paper utilizes a combination of datasets for meta-training and evaluation:
- Meta-Training: A collection of 39 benchmark datasets, comprising 18 from the DAMI library and 21 from the ODDS library. These datasets are used during the offline phase to train the PPE and MSFs.
- Evaluation: The same 39 datasets are used for evaluating HPOD and its baselines. The evaluation is performed using a leave-one-out cross-validation strategy, where each dataset is used as a test set once, and the remaining 38 are used for meta-training.
No new benchmark datasets were introduced; instead, existing, well-established datasets were leveraged.
Key Insights and Ideas
- Meta-Learning for Unsupervised HPO: The core innovation is applying meta-learning to bridge the gap between supervised HPO techniques (like SMBO) and the label-scarce nature of unsupervised OD. By learning from past experiences on labeled data, HPOD can effectively optimize HPs for unlabeled tasks.
- Proxy Performance Evaluation: The development of a PPE that can accurately predict OD performance using only dataset characteristics and internal signals (IPMs) is crucial. This bypasses the need for ground truth labels during the online optimization phase.
- Continuous HP Search: Unlike many existing unsupervised OD model selection methods that rely on discrete HP sets, HPOD's ability to handle continuous HP spaces is a significant advantage, especially for complex models with many HPs.
- Internal Performance Measures (IPMs) as Signals: The paper highlights the utility of IPMs, which are weak but useful signals derived from outlier scores, as supplementary features for the PPE. This demonstrates that even without ground truth, valuable information can be extracted from the data and the model's outputs.
- Importance of HPO: The results consistently show that default HPs are often far from optimal, and even simple baselines like random search can be outperformed by a structured approach like HPOD, underscoring the critical need for effective HPO in OD.
Key Results
HPOD demonstrates significant improvements across various OD algorithms and datasets:
- RAE (Deep Model): HPOD achieved a 37% improvement in average normalized AP rank compared to HPs selected by minimal reconstruction loss.
- LOF (Shallow Model): HPOD provided a 58% improvement in average normalized AP rank compared to default HPs.
- iForest (Ensemble Model): HPOD achieved a 66% improvement in average normalized AP rank compared to default HPs.
- Generality: HPOD consistently outperformed diverse baselines, including simple methods (Default, Random, MinLoss) and complex meta-learning-based unsupervised OD model selection methods (HyperEns, GB, ISAC, AS, MetaOD), across both discrete and continuous HP spaces.
- Statistical Significance: Pairwise statistical tests (Wilcoxon signed rank test) confirmed that HPOD is statistically better than most baselines across RAE, LOF, and iForest.
Limitations and Future Work
- Computational Cost: The offline meta-training phase is resource-intensive. While amortized over many tasks, it's a significant upfront investment. The online phase, while efficient, still requires IPM calculation, adding computational demands.
- Future Work: The authors suggest enhancing computational efficiency by refining approximation methods for model building and exploring more efficient performance evaluation algorithms. They also propose future work on joint algorithm selection and continuous hyperparameter optimization for unsupervised OD.
Summary updated