The Case for Simple Baselines in Anomaly Detection

22 concepts19 papers

In machine learning, there is a pervasive assumption that architectural complexity scales with performance. When faced with intricate, high-dimensional data like multivariate time series, medical images, or out-of-distribution video frames, the default research trajectory often leans toward heavy, non-linear deep learning models—such as Transformers, Generative Adversarial Networks (GANs), and complex autoencoders.

However, a growing body of rigorous benchmarking literature reveals a stark complexity-performance mismatch. Across multiple domains, simple linear projections, classical statistical models, and even untrained neural networks frequently match or outperform state-of-the-art deep architectures. This phenomenon suggests that the added representational capacity of highly complex models is often unjustified by the underlying characteristics of current datasets, creating an "illusion of progress" driven by flawed evaluation metrics rather than genuine algorithmic superiority.

The Illusion of Progress

The field of unsupervised Time Series Anomaly Detection (TSAD) serves as a primary case study for this complexity trap. As deep learning architectures have grown more sophisticated, their actual performance gains over fundamental baselines have largely vanished when subjected to strict, unbiased evaluations.

M. Saquib Sarfraz et al. and Dennis Wagner et al. independently exposed this trend, demonstrating that basic linear methods and standard baseline architectures (such as LSTM-based autoencoders) consistently achieve performance parity with modern generative and attention-based models. When high-capacity models fail to outperform minimalist methods, it indicates that the models are not effectively utilizing their non-linear modeling capacity to solve the underlying task.

This pattern is not isolated to time series. In medical imaging, Felix Meissen et al. challenged the state-of-the-art in semi-supervised anomaly segmentation for brain MRIs. They discovered that a simple intensity thresholding baseline matches or outperforms complex generative models. Because many pathologies inherently manifest as straightforward signal anomalies (such as bright pixels in FLAIR sequences), the computational overhead and optimization complexity of deep generative models often introduce more noise than value.

Why Simple Models Succeed

To understand why basic models perform so well, we must look at how anomalies are structurally defined within standard datasets. Anomalies are fundamentally deviations from a normal subspace or expected temporal pattern. If these patterns are predominantly linear or low-dimensional, complex non-linear models overfit to the noise rather than capturing the underlying distribution.

Consider a simple Ordinary Least Squares (OLS) linear regression model utilizing lagged features. Aleksandr Yugay et al. demonstrated that such a model can achieve state-of-the-art accuracy and robustness in time series anomaly detection while requiring a fraction of the computational resources of deep learning alternatives. The OLS model succeeds because it directly and efficiently captures the autoregressive dependencies of the sequence without the risk of optimization failure or representation collapse inherent in deep networks.

Similarly, Principal Component Analysis (PCA) reconstruction error serves as an incredibly potent baseline. By projecting data onto a lower-dimensional linear subspace, PCA discards high-frequency noise. When an anomalous data point appears, it fails to map cleanly into this normal subspace, resulting in a high reconstruction error. As M. Saquib Sarfraz et al. argue, this basic geometric mechanism is often all that is required to separate normal states from anomalies in current benchmarks.

Flawed Metrics Masking the Truth

If simple models are so effective, how did the research community conclude that highly complex architectures were necessary? The answer lies in systemic evaluation flaws, most notably the "Point Adjustment" (PA) protocol.

Under the traditional PA protocol, if a model correctly identifies a single point within an entire contiguous anomalous segment, the protocol automatically adjusts the model's predictions to mark the entire segment as correctly detected. Siwon Kim et al. exposed the danger of this shortcut by showing that completely untrained, randomly initialized neural networks and raw input magnitudes can achieve state-of-the-art scores when PA is applied. Because random guessing or simple distance-based heuristics are highly likely to hit at least one point in a long anomaly window, PA artificially inflates the F1-scores of high-capacity models, masking their inability to precisely localize anomalies.

When benchmarks remove these evaluation artifacts and enforce strict point-wise metrics, the apparent superiority of deep learning evaporates. Qinghua Liu et al. introduced the TSB-AD benchmark to provide a reliable, large-scale evaluation framework. Their findings confirm that classical statistical methods frequently outperform complex neural networks on univariate sequences, proving that the representational power of deep networks can lead to lower accuracy compared to simpler subspace approaches.

Empirical Evidence Across Domains

The robustness of low-complexity methods spans across diverse data modalities and problem formulations:

  • Univariate Time Series: Large-scale comparative studies by Yan Zhao et al. and Xiangfei Qiu et al. show that traditional non-learning and shallow machine learning algorithms consistently match deep learning performance while offering superior computational efficiency and lower hardware requirements. Ferdinand Rewicki et al. highlighted that classical methods like MDI and MERLIN generally outperform deep learning approaches across various anomaly types on the UCR Anomaly Archive. Nesryne Mejri further demonstrated that simple, deterministic algorithms excel particularly when data contains non-trivial patterns that deep models struggle to optimize reliably.
  • Multivariate Time Series: Even when dealing with multiple interacting variables, the complexity of modern architectures remains questionable. Julien Audibert et al. argue that the "bigger is better" mentality has trapped multivariate research in an engineering loop, producing models that are difficult to train and interpret without yielding superior performance over traditional statistical baselines.
  • Out-of-Distribution (OOD) Detection: In computer vision, OOD detection often relies on complex auxiliary training regimes or heavy hyperparameter tuning. Dan Hendrycks et al. challenged this by introducing MaxLogit, a simple scoring mechanism based on raw model outputs. MaxLogit consistently outperforms specialized, high-complexity OOD detection methods across multiclass, multi-label, and segmentation tasks at scale, proving that basic logit-based baselines are often sufficient.
  • Tabular and Few-Shot Settings: When analyzing granular design choices, Minqi Jiang et al. found that basic Multi-Layer Perceptron (MLP) architectures provide more stable and competitive performance than Transformers or residual networks, especially when labeled data is scarce. For few-shot anomaly detection, Niamh Belton et al. showed that low-complexity Siamese networks paired with a robust loss function can outperform heavier backbones while drastically reducing training time. Even model-free distance approaches, such as the $k$-nearest neighbors ($k$-NN) framework proposed by Vladislav Ishimtsev et al., can match or exceed deep networks on non-stationary data streams when properly calibrated with statistical frameworks like Conformal Prediction.

Moving Forward

The collective evidence from these researchers does not imply that deep learning has no place in anomaly detection. Rather, it serves as a methodological call to action. Future research must resist the temptation of architectural over-engineering. To demonstrate true progress, newly proposed high-capacity models must be consistently and rigorously benchmarked against strong, well-calibrated linear and classical baselines. If a simple OLS model or a PCA projection can solve the task equally well, the computational overhead, energy consumption, and interpretability trade-offs of a massive neural network remain fundamentally unjustified.

Key Foundational Work

Conformal k-NN Anomaly Detector for Univariate Data Streams (2017) — contributed 1 concept elements to this direction.
Challenging Current Semi-supervised Anomaly Segmentation Methods for Brain MRI (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unjustified_architectural_complexity_6b2e5301,
  title = {unjustified architectural complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unjustified-architectural-complexity/rdr_6b2e5301e2ce4197a34eb92488b04b18},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_unjustified_architectural_complexity_6b2e5301_v1,
  title = {unjustified architectural complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unjustified-architectural-complexity/rdr_6b2e5301e2ce4197a34eb92488b04b18/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake