Evaluating Language Model Features for Zero-Shot Time-Series Anomaly Detection Under Isolation Assumption

Romain Hermary, Nesryne Mejri, Djamila AouadaarXiv 2026 · arXiv

TL;DR

Can Large Language Models (LLMs) detect anomalies in complex time-series data without any specific training? This paper shows that by treating pre-trained LLMs as zero-shot feature extractors and pairing them with an Isolation Forest, they can outperform specialized time-series models. Surprisingly, the authors find that the Transformer architecture's inherent structure is often more important than the language knowledge it was trained on.

What Problem Does This Paper Address?

Multivariate Time-Series Anomaly Detection (MTSAD) is a critical but difficult task. It’s mostly unsupervised because real-world anomalies are rare and hard to label. While researchers have tried adapting LLMs for this, most existing methods rely on "proxy tasks" like forecasting the next value or reconstructing the input.

The authors argue that as models get more powerful, they might become too good at these tasks—faithfully forecasting or reconstructing the anomalies themselves, which makes them impossible to detect via error residuals. Furthermore, common tokenization methods (like patching) often blur the fine-grained local details needed to spot subtle anomalies. This paper investigates whether LLMs can provide useful representations for anomaly detection directly in a zero-shot setting.

Methods

The researchers propose a pipeline that uses LLMs purely as feature extractors:

  1. Step-Wise Tokenization: Instead of grouping time-steps into patches, they map each individual time-step to a single token to preserve local information. They tested three schemes: Padding (adding zeros), Interpolation (rescaling the input), and Closest Vocabulary (quantizing time-steps to the nearest text token embedding).
  2. Zero-Shot Feature Extraction: The tokenized sequence is passed through a frozen, pre-trained LLM (such as GPT-2, Mistral, or Qwen). The high-dimensional embeddings from the model's internal layers are flattened into a fixed-size feature vector.
  3. Isolation Forest: They apply an Isolation Forest—an unsupervised algorithm that isolates samples by partitioning the feature space—to these extracted features. The core idea is that anomalies are "few and different" and thus isolated more quickly than normal data.
The proposed pipeline: Tokenization, Zero-Shot Feature Extraction via LLM, and Transferability Quantification using an Isolation Forest.

Datasets

The study evaluated performance on four standard multivariate time-series benchmarks:

  • PSM (Pooled Server Metrics): Metrics collected from multiple server nodes.
  • PUMP (PUMP Fillage): Industrial sensor data from a water pump system.
  • SWaT (Secure Water Treatment): Data from a physical water treatment testbed.
  • WaDi (Water Distribution): An extension of SWaT representing a larger distribution network.

Key Insights and Ideas

  • Architecture is the Hero: The most surprising finding is that randomly initialized LLMs often performed as well as, or even better than, pre-trained ones. This suggests that the Transformer's attention-based architecture has a natural "inductive bias" that helps separate unusual patterns from normal ones, regardless of the language knowledge it holds.
  • Don't Over-Quantize: The "Closest Vocabulary" method, which forces continuous time-series data into discrete text tokens, consistently performed the worst. Non-disruptive methods like Interpolation, which preserve local information, were far more effective.
  • Bigger Isn't Always Better: In this zero-shot setting, smaller models (like Qwen 0.5B) frequently outperformed their much larger 7B counterparts.
  • Proxy Task Failure: The results confirmed that standard forecasting and reconstruction objectives are often suboptimal for anomaly detection, as high-capacity models can "absorb" the very signals they are meant to detect.

Key Results

  • Superior Zero-Shot Performance: LLM-based features consistently improved detection over the "Raw" data baseline across all datasets.
  • Outperforming Specialized Models: The zero-shot LLM approach often surpassed specialized Time-Series Foundation Models (TSFMs) like MOMENT and Chronos. For instance, using Chronos as a feature extractor with an Isolation Forest improved its performance on the SWaT dataset by over 40 AUROC points compared to using its native forecasting strategy.
  • Significant Gains from Architecture: Randomly initialized models achieved gains of up to 29 points in AUPR (Area Under Precision-Recall) over simple token-based classification on the SWaT dataset.
Figure 3: AUROC results of the studied LLMs on different datasets, with Interpolation. We show the results difference with the baseline (i.e., Raw path) and the results achieved by each LLM, i.e., ∆(LLM, Raw). The average baseline results are indicated on the slices sides and are expected results on the 0 ring.

Limitations and Future Work

The authors acknowledge that their study is limited to four industrial datasets and relies on the "isolation assumption" (that anomalies are rare and structurally distinct). Future work could involve testing a wider variety of anomaly types and unsupervised classification methods beyond the Isolation Forest. They also highlight a major open challenge: developing better tokenization strategies that can bridge the gap between continuous time-series and discrete language representations without losing the fine-grained local structure essential for anomaly detection.

Summary updated

Abstract

Time-Series Anomaly Detection (TSAD) is a fundamentally challenging task: it is unsupervised, requires high local precision capabilities, and suffers from limited labelled data. Recent work has explored adapting Large Language Models (LLMs) to time-series via cross-modal transfer. However, existing approaches are poorly suited to Multivariate TSAD (MTSAD), as tokenisation often obscures fine-grained temporal and variables structures, and most methods optimise auxiliary objectives such as forecasting or reconstruction. We posit that these surrogate objectives become unreliable as model capacity increases. Highly expressive models risk reconstructing or forecasting anomalies, undermining the foundations of both strategies. Instead, we propose to use LLMs purely as feature extractors and directly analyse their induced feature spaces. In this work, investigate the ability of LLMs to provide useful representations for MTSADs in a zero-shot setting through classification. To this end, we study the separability of the extracted features, under the assumption that anomalous features should be isolated from normal ones. We analyse their temporal and contextual discriminative ability using simple step-wise tokenisation schemes, with one time-step per token, which preserve fine-grained local information. Their usability for MTSAD is assessed using an unsupervised Isolation Forest, enabling a direct evaluation of the resulting representation quality. We further investigate whether the observed performance can be attributed to the language-related pre-training of the LLMs or arises independently of it. Our results provide empirical insight into the ability of LLMs to represent multivariate time-series for anomaly detection and highlight broader applications of cross-modal foundation models.
BibTeX
@misc{hermary2026evaluating,
  author = {Romain Hermary and Nesryne Mejri and Djamila Aouada},
  title = {{Evaluating Language Model Features for Zero-Shot Time-Series Anomaly Detection Under Isolation Assumption}},
  year = {2026},
  eprint = {2611.07720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07720},
}
Figure