TL;DR
Can Large Language Models (LLMs) detect anomalies in complex time-series data without any specific training? This paper shows that by treating pre-trained LLMs as zero-shot feature extractors and pairing them with an Isolation Forest, they can outperform specialized time-series models. Surprisingly, the authors find that the Transformer architecture's inherent structure is often more important than the language knowledge it was trained on.
What Problem Does This Paper Address?
Multivariate Time-Series Anomaly Detection (MTSAD) is a critical but difficult task. It’s mostly unsupervised because real-world anomalies are rare and hard to label. While researchers have tried adapting LLMs for this, most existing methods rely on "proxy tasks" like forecasting the next value or reconstructing the input.
The authors argue that as models get more powerful, they might become too good at these tasks—faithfully forecasting or reconstructing the anomalies themselves, which makes them impossible to detect via error residuals. Furthermore, common tokenization methods (like patching) often blur the fine-grained local details needed to spot subtle anomalies. This paper investigates whether LLMs can provide useful representations for anomaly detection directly in a zero-shot setting.
Methods
The researchers propose a pipeline that uses LLMs purely as feature extractors:
- Step-Wise Tokenization: Instead of grouping time-steps into patches, they map each individual time-step to a single token to preserve local information. They tested three schemes: Padding (adding zeros), Interpolation (rescaling the input), and Closest Vocabulary (quantizing time-steps to the nearest text token embedding).
- Zero-Shot Feature Extraction: The tokenized sequence is passed through a frozen, pre-trained LLM (such as GPT-2, Mistral, or Qwen). The high-dimensional embeddings from the model's internal layers are flattened into a fixed-size feature vector.
- Isolation Forest: They apply an Isolation Forest—an unsupervised algorithm that isolates samples by partitioning the feature space—to these extracted features. The core idea is that anomalies are "few and different" and thus isolated more quickly than normal data.
Datasets
The study evaluated performance on four standard multivariate time-series benchmarks:
- PSM (Pooled Server Metrics): Metrics collected from multiple server nodes.
- PUMP (PUMP Fillage): Industrial sensor data from a water pump system.
- SWaT (Secure Water Treatment): Data from a physical water treatment testbed.
- WaDi (Water Distribution): An extension of SWaT representing a larger distribution network.
Key Insights and Ideas
- Architecture is the Hero: The most surprising finding is that randomly initialized LLMs often performed as well as, or even better than, pre-trained ones. This suggests that the Transformer's attention-based architecture has a natural "inductive bias" that helps separate unusual patterns from normal ones, regardless of the language knowledge it holds.
- Don't Over-Quantize: The "Closest Vocabulary" method, which forces continuous time-series data into discrete text tokens, consistently performed the worst. Non-disruptive methods like Interpolation, which preserve local information, were far more effective.
- Bigger Isn't Always Better: In this zero-shot setting, smaller models (like Qwen 0.5B) frequently outperformed their much larger 7B counterparts.
- Proxy Task Failure: The results confirmed that standard forecasting and reconstruction objectives are often suboptimal for anomaly detection, as high-capacity models can "absorb" the very signals they are meant to detect.
Key Results
- Superior Zero-Shot Performance: LLM-based features consistently improved detection over the "Raw" data baseline across all datasets.
- Outperforming Specialized Models: The zero-shot LLM approach often surpassed specialized Time-Series Foundation Models (TSFMs) like MOMENT and Chronos. For instance, using Chronos as a feature extractor with an Isolation Forest improved its performance on the SWaT dataset by over 40 AUROC points compared to using its native forecasting strategy.
- Significant Gains from Architecture: Randomly initialized models achieved gains of up to 29 points in AUPR (Area Under Precision-Recall) over simple token-based classification on the SWaT dataset.
Limitations and Future Work
The authors acknowledge that their study is limited to four industrial datasets and relies on the "isolation assumption" (that anomalies are rare and structurally distinct). Future work could involve testing a wider variety of anomaly types and unsupervised classification methods beyond the Isolation Forest. They also highlight a major open challenge: developing better tokenization strategies that can bridge the gap between continuous time-series and discrete language representations without losing the fine-grained local structure essential for anomaly detection.
Summary updated