Automatically configuring an anomaly detector means choosing its architecture, its training settings and the features it looks at. Without labels, nobody can tell which configuration is best, so "find the best one" is the wrong target. We propose a different target that can be measured without labels: find a small set of configurations whose mistakes are not shared. Our method samples detectors from a joint space of architectures, hyperparameters and feature subsets. It keeps a candidate only if its anomaly ranking is reproducible on refits and differs from the rankings of the detectors already kept. It then averages the survivors.
This matters most on high-dimensional tabular data with many redundant or irrelevant columns, which is where automated anomaly detection currently breaks. Deep unsupervised detectors lose to shallow ones on tabular data because they cannot be tuned without labels and are sensitive to noise features, as ADBench showed. Many detectors also lock onto the same dominant directions of variation, and those directions are often nuisance rather than anomaly signal: Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection found that high-variance components mostly carry background variation and the anomaly signal sits in scattered low-variance ones. When every ensemble member relies on the same nuisance directions, averaging them gives no benefit. Our bet is that redundancy between detectors can be detected without labels and early in training, even though detector quality cannot.
Core Claim
In unsupervised anomaly detection, a search over architecture, hyperparameters and feature subspaces should pursue diversity weighted by a quality score rather than accuracy. Both ingredients can be estimated without labels.
Quality is measured by stability. We refit a candidate on two bootstrap subsamples and take the Spearman correlation of the anomaly rankings it assigns to a shared reference set of unlabeled points. A detector that responds to real structure should rank points reproducibly. A detector that responds to noise should not.
Redundancy is measured by similarity between two candidates' rankings on that same reference set, optionally mixed with how much their feature subsets overlap.
The two measures combine in a determinantal point process (DPP). A DPP is a probability model over subsets that favors sets whose members are individually good and mutually dissimilar, by weighting each set by the determinant of a kernel matrix. The kernel for candidates $i$ and $j$ is
$$ L_{ij} = q_i \, q_j \left[\alpha\, \rho_{ij} + (1-\alpha)\, J_{ij}\right], $$
where $q_i$ is the bootstrap rank stability of candidate $i$, $\rho_{ij}$ is the rank correlation of the two candidates' scores on the reference set, $J_{ij}$ is the Jaccard overlap of their feature subsets, and $\alpha \in [0,1]$ trades off output redundancy against input redundancy. The ensemble is the size-$k$ subset that approximately maximizes $\log\det L_S$, found by greedy selection.
The two measures do separate jobs. Stability rejects detectors that are diverse only because they are noisy: a random scorer is maximally diverse but unstable. Diversity then prevents the ensemble from filling up with many stable copies of one nuisance-driven view.
The second, testable part of the claim is about cost. We hypothesize that the similarity $\rho_{ij}$ between candidates settles after a few training epochs, whereas unsupervised quality signals such as reconstruction loss are known to mislead at low fidelity. If that holds, the rule that decides which candidates get more training budget can use marginal diversity gain instead of loss.
Relation to Prior Work
Three systems each cover two of the three directions, and each supplies a piece we reuse or contrast against.
HPOD tunes hyperparameters of fixed detector templates (a recurrent autoencoder, the Local Outlier Factor and Isolation Forest) without labels. It uses a performance predictor meta-learned over 39 labeled datasets. It returns a single configuration, performs no architecture search and treats the feature space as fixed. We borrow its leave-one-dataset-out protocol and its idea of internal performance measures, meaning label-free signals computed from a detector's own outputs. We differ in two ways: we avoid depending on a meta-database, and we select a set rather than a point.
AutoOD jointly searches neural architectures and outlier definitions (density, cluster, centroid, reconstruction) with a reinforcement-learning controller. It also outputs one child detector, works on raw inputs, and needs a validation signal. Our search space is similar in spirit, but the output and the selection criterion differ.
ODEFS couples feature weighting with a bagged linear ranking detector. It is the strongest evidence that embedded feature selection helps here: on the Advertisements dataset it cut 1,555 features to 48 and raised AUC from 0.722 to 0.906. However, it has a fixed linear model and fixed hyperparameters, and its bags differ only in data subsamples. They are not selected for diversity.
Our selection machinery comes from two DPP papers. A determinantal point process for column subset selection gives projection DPPs that sample non-redundant feature columns. We use it to propose feature subspaces for candidates, instead of random subsets or leading principal components. DNS uses a size-$k$ DPP over the representations of ensemble members, compared by centered kernel alignment, to stop reinforcement-learning critic ensembles from collapsing into redundant copies. It uses less than 50% of the compute of the full ensemble. We carry that anti-collapse idea into anomaly detection with a label-free quality term, which DNS does not need because critic training has a supervised target.
Two papers constrain the design. SEAD shows that plain mean or max aggregation degrades when noisy or redundant members are in the pool. It also shows that a label-free rarity prior can reweight members online. This is both a warning (pool composition matters) and a competing remedy: reweighting rather than selecting. Auto-PyTorch reports that post-hoc ensembles built from the top configurations of a supervised search overfit the validation split. That is why we avoid a top-$M$ rule, which would pick near-duplicates.
Research Program
Candidate space. We will build a joint space over three things. The first is detector family and architecture: autoencoders with searched depth, width, bottleneck size and activation, alongside Isolation Forest, Local Outlier Factor and one-class SVM, each with its own hyperparameters. The second is training hyperparameters: learning rate, weight decay, epochs and a contamination assumption. The third is a feature subspace, drawn from a projection DPP over the standardized data columns with a searched subspace size, and including the full feature set as one option.
Search loop. The loop is successive halving, where candidates are trained cheaply and only the most promising get more budget.
- Sample a batch of candidates and train each at low fidelity: few epochs, or a data subsample for the shallow models.
- Score a fixed reference set of unlabeled points and compute each candidate's stability and its similarity to the current ensemble.
- Promote the candidates with the largest marginal gain in $\log\det L_S$ to the next training budget.
- At the end, take the greedy size-$k$ selection and combine members by averaging their rank-normalized scores.
The method's own settings ($k$, $\alpha$, the reference-set size and the fidelity schedule) are fixed once on a small meta-split of datasets that is disjoint from all test datasets. No test labels are used anywhere.
Low-fidelity diagnostic. For every dataset we will measure how well rankings at low fidelity predict rankings at full training, using Kendall's $\tau$, a rank-agreement measure. We compute it separately for pairwise similarity $\rho_{ij}$, for stability $q_i$, and for reconstruction loss. The claim that the method is cheap depends on the first $\tau$ being much higher than the last.
Baselines
The naive reference is Isolation Forest with default settings, together with the best default-configured single detector per benchmark drawn from the PyOD families. Every method must beat these to be worth its cost. Where the file gives no aggregate figure, we will report our own. We found no reported result on Lacuna for the median unsupervised AUCROC of default Isolation Forest across ADBench.
The strongest search competitor is HPOD applied to each detector family, keeping the best predicted configuration. It is the only published method that tunes anomaly detectors without labels. We will run it under the same leave-one-dataset-out protocol on the ODDS/DAMI datasets it was built for, with the same number of trained configurations as our search. The file reports its gains only relative to other choices: a 58% improvement in average normalized average-precision rank for the Local Outlier Factor over defaults, and 37% for the recurrent autoencoder over configurations chosen by minimal reconstruction loss, both from HPOD. The Isolation Forest comparator is truncated in the file. No absolute AP or AUC figure is available, so we will reproduce HPOD's absolute numbers ourselves. Beating it means a better average rank in average precision across held-out datasets, with a paired Wilcoxon test.
The strongest high-dimensional competitor is ODEFS on its 12 high-dimensional datasets. The file reports that ODEFS outperformed competitors on 10 of the 12 and reached AUC 0.906 on Advertisements. We will match its protocol, and beating it on Advertisements means exceeding 0.906 AUC with no labels used for selection. Against SEAD we will compare on its non-streaming ODDS subset, where it reports an average rank of 5.07 in its own comparison. We will rerun SEAD within our comparison rather than reuse that rank.
The pool baselines use exactly the same trained candidates as our method and differ only in which members are kept and how they are combined. They are: a random size-$k$ subset with mean rank averaging, the whole pool with mean rank averaging, the whole pool with SEAD-style label-free weights, and the top-$k$ candidates by stability alone.
The ablations isolate the key ingredient. They are: diversity without the quality term ($q \equiv 1$), which should admit noise detectors; output similarity only ($\alpha = 1$); feature overlap only ($\alpha = 0$); random rather than DPP-proposed feature subspaces; and loss-based rather than diversity-based promotion between fidelity levels.
Why not just…?
Why not just average a large random pool, as in feature bagging? This is the most serious threat. A big random pool spans the joint space by brute force, and rank averaging is cheap. SEAD's evidence that mean aggregation breaks when noisy members are present argues against it, but that evidence comes from injected noise models, not from members that are redundant through shared features. The pool baselines above answer the question directly, because they use the same candidates. If random-subset or whole-pool averaging matches our selection across the redundancy sweep described below, the diversity machinery adds nothing, and we would report that as the finding.
Why not just extend HPOD's meta-learned predictor to cover architectures and pick the single best configuration? A single configuration is limited by the no-free-lunch problem across anomaly types that motivates ensembles in the first place. HPOD's predictor also depends on target datasets resembling its 39 meta-training datasets, and heavily redundant feature spaces are exactly where that resemblance may fail. Our comparison splits results by the ADBench irrelevant-feature ratio. It will show whether HPOD's advantage shrinks as redundancy grows while ours holds. If HPOD wins even at high redundancy, a better single model beats a diverse set, and we would say so.
Why not just reweight the pool online with SEAD instead of selecting from it? Reweighting can suppress bad members, but it does not remove redundant good ones. Ten stable copies of one nuisance-driven view keep substantial weight under a rarity prior. SEAD weighting over the full pool is already one of our baselines, and we will also run SEAD weights on top of our selected set. If the two are complementary, the combination should beat both.
Why not just run principal component analysis first and then any detector? Gula & Bertoldo show that the high-variance components discard the anomaly-bearing ones, and ODEFS beat sequential selection baselines. We include a leading-components baseline for completeness.
Proposed Experiments
The main experiments use ADBench's 57 tabular datasets in the unsupervised setting, reporting AUCROC and average precision. The ODDS/DAMI datasets are used for the HPOD comparison, and the ODEFS high-dimensional datasets, including Advertisements, for the feature-redundancy regime. We also use ADBench's four anomaly types: local, global, dependency and clustered. We report results by type, because clustered and duplicated anomalies are where density-based members fail together; ADBench reports a -16.43% median drop under duplication. Compute is matched to a fixed count of trained configurations per dataset, applied to every search-based method.
First decisive check: the redundancy sweep. Using ADBench's irrelevant-feature injection, we raise the fraction of noise columns and track the gap in average precision between our selection and random-subset averaging. The hypothesis predicts that this gap widens as redundancy increases. A flat gap falsifies the central mechanism.
Second decisive check: early redundancy estimates. We test whether low-fidelity similarity $\rho_{ij}$ predicts the full-training value with high Kendall's $\tau$ while reconstruction loss does not. If similarity also fails at low fidelity, the method still stands, but only at full training cost, and the claim that it is cheap is withdrawn.
Third decisive check: the quality term. Removing $q$ should visibly admit low-stability members and hurt accuracy. This would confirm that stability is what separates useful disagreement from noise.
We will also report ensemble size against accuracy, and the feature subsets chosen on Advertisements, to see whether they converge toward ODEFS's compact subspace or stay spread out.
References
- ADBench: Anomaly Detection Benchmark
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- HPOD: Hyperparameter Optimization for Unsupervised Outlier Detection
- AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
- Outlier Detection Ensemble with Embedded Feature Selection
- A determinantal point process for column subset selection
- DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learning
- SEAD: Unsupervised Ensemble of Streaming Anomaly Detectors
- Auto-Pytorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
- PyOD: A Python Toolbox for Scalable Outlier Detection