Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection

Ying Fu Lim, Jiawen Zhu, Guansong Pang

Introduction

In the landscape of modern distributed systems, log data serves as the primary diagnostic tool for identifying system failures and security breaches. However, the sheer volume and velocity of logs generated by high-performance computing clusters and cloud environments make manual inspection impossible. This has given rise to Log Anomaly Detection (LAD), a field focused on identifying "needles in a haystack"—unusual patterns that deviate from normal system behavior.

While traditional deep learning architectures like LSTMs, CNNs, and specialized transformers (e.g., LogBERT) have been the standard, they face significant hurdles. These models often require training from scratch for every new system and struggle with the evolving vocabulary of logs. Large Language Models (LLMs) like Llama-3 offer a promising alternative due to their pre-trained semantic understanding, but their deployment is hindered by the high cost of full fine-tuning and the unreliability of zero-shot prompting.

Methodology: Efficient Adaptation

This research explores the use of Parameter-Efficient Fine-Tuning (PEFT) to adapt LLMs for LAD without the computational burden of full-model updates. The study focuses on three backbone models: RoBERTa (125M), GPT-2 (117M), and Llama-3 (8B).

Two primary PEFT techniques are compared:

  1. Low-Rank Adaptation (LoRA): This method injects trainable low-rank matrices into the model's self-attention layers. It approximates the necessary weight updates while keeping the original pre-trained weights frozen.
  2. Representation Fine-tuning (ReFT): A more recent approach that leaves the model weights entirely untouched. Instead, it applies "interventions" to the hidden representations (internal vectors) as data flows through the model layers.

The Detection Pipeline

The workflow begins by parsing raw logs into structured templates and grouping them into sequences using a sliding window approach. These sequences are fed into the PEFT-adapted LLM. A classification head then analyzes the final hidden state—the [CLS] token for RoBERTa or the final token for Llama-3—to generate an anomaly score.

Experimental Setup and Datasets

The models were evaluated against four gold-standard public log datasets representing different system architectures:

  • HDFS: Hadoop Distributed File System logs.
  • BGL: BlueGene/L supercomputer logs.
  • Spirit & Thunderbird: Logs from high-performance computing clusters.

These datasets provide a rigorous benchmark for comparing the proposed LLM-driven approach against traditional supervised and unsupervised baselines.

Key Insights

1. The Superiority of Scale

Llama-3 (8B) consistently outperformed smaller models across all datasets. Its extensive pre-training allows it to capture complex, subtle semantic patterns in log data that smaller, specialized models often miss. This suggests that the general-purpose knowledge embedded in large models is highly transferable to the domain-specific task of log analysis.

2. ReFT vs. LoRA

A pivotal finding of this study is that ReFT outperformed LoRA in 9 out of 12 comparisons. Modifying internal representations directly appears to be a more effective strategy for LAD than modifying model weights, particularly when training data is limited. While ReFT's training time can increase with higher ranks, it achieves peak performance at very low ranks (e.g., rank 8), making it highly efficient for practical use.

3. Sample Efficiency and Robustness

LLMs adapted with PEFT demonstrated remarkable sample efficiency. Llama-3-ReFT achieved high F1 scores even when trained on only 10% of the available data, a scenario where traditional models typically fail. Furthermore, the models showed high robustness to "log evolution." When log keywords were replaced with synonyms to simulate software updates, the PEFT-adapted models maintained stable performance, whereas traditional methods often require retraining.

Performance Results

The LLM-driven models established new benchmarks for detection accuracy. On the BGL dataset, Llama-3-ReFT achieved an F1-score of 0.9690, significantly surpassing state-of-the-art baselines like LogRobust and CNN-based detectors.

However, the "final boss" of LAD—cross-dataset generalization—remains a challenge. When a model trained on the Spirit dataset was tested on BGL or HDFS, performance dropped significantly. Despite this, ReFT-based Llama-3 still outperformed LoRA-based versions, which often saw their F1 scores drop to near zero in cross-system scenarios.

Limitations and Future Directions

While PEFT makes training feasible, inference latency remains a concern. Running an 8-billion parameter model for real-time monitoring of millions of logs per second is computationally expensive compared to lightweight LSTMs. Future research should focus on model compression techniques, such as distillation or quantization, to bridge the gap between high-accuracy LLMs and the speed requirements of production environments.

Additionally, while ReFT showed promise in generalization, further work is needed to create models that can transition between different system architectures without requiring local fine-tuning.

Summary updated

BibTeX
@inproceedings{lim2025adapting,
  author = {Ying Fu Lim and Jiawen Zhu and Guansong Pang},
  title = {{Adapting Large Language Models for Parameter-Efficient Log Anomaly Detection}},
  booktitle = {PAKDD},
  year = {2025},
  doi = {10.1007/978-981-96-8298-0\_26},
}
Figure