BERT-LogAnom: Enhancing Log Anomaly Detection with Gated Residual BiLSTM and Dynamic Thresholding

Xi Ning Lu, Shufan An, Jingmei Chen, Zhan Shu, Weiping Wang, Runyi Qi, Yapeng DiaoElectronics 2026 · DOI

TL;DR

This paper introduces BERT-LogAnom, a novel unsupervised framework for detecting anomalies in system logs. It combines BERT's contextual understanding with a specialized BiLSTM for sequential patterns and a dynamic thresholding mechanism to adapt to changing system behavior, achieving state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Modern software systems are increasingly complex, generating massive amounts of log data. Detecting anomalies in these logs is crucial for maintaining system reliability and diagnosing faults. However, existing methods struggle with two main challenges:

  1. Capturing Sequential Dependencies: Anomalies often manifest as subtle deviations in the order or sequence of events, which traditional methods might miss.
  2. Robustness to Distributional Changes: System logs are dynamic; workloads shift, configurations change, and software evolves. This "concept drift" makes static anomaly detection thresholds unreliable, leading to false positives or missed anomalies.

Methods

BERT-LogAnom tackles these challenges with a multi-component architecture:

  • BERT-based Contextual Encoding: Leverages BERT's powerful self-attention mechanism to learn global semantic representations of log sequences, understanding the context of individual log messages.
  • Gated Residual BiLSTM (GR-BiLSTM): This novel module is added on top of BERT's output. It explicitly models bidirectional temporal dependencies (forward and backward) within log sequences. The "gated" aspect allows it to dynamically emphasize events more indicative of anomalies, and the "residual" connection ensures BERT's global context isn't lost.
  • Self-Supervised Learning Objectives: To learn from unlabeled data, the model is trained using two objectives:
  • Masked Log Key Prediction (MLKP): Similar to BERT's masked language modeling, it predicts masked log templates, forcing the model to understand sequential patterns.
  • Volume Hypersphere Minimization (VHM): Encourages normal log representations to cluster tightly in a feature space, making anomalous patterns (which fall outside this cluster) easier to detect.
  • Dynamic Thresholding Prediction Module (DTPM): This is a key innovation for handling concept drift. It uses a two-layer approach:
  • Fast Adaptation (EWMA): Tracks short-term fluctuations in anomaly scores to quickly respond to sudden shifts.
  • Slow Calibration (Gated Neural Network): Handles long-term distribution drift by learning a calibration factor. This allows the anomaly detection threshold to adapt over time without needing new labeled data.

Datasets

The framework was evaluated on three publicly available, widely used benchmark datasets:

  • HDFS: Logs from Hadoop Distributed File System jobs, labeled using handcrafted rules.
  • BGL: Logs from the BlueGene/L supercomputer system, with anomalies identified by alert messages.
  • Thunderbird: A large-scale log dataset from a supercomputer system.

The paper notes that for these datasets, log sequences were constructed using session (block) identifiers for HDFS, and time-based sliding windows for BGL and Thunderbird.

Key Insights and Ideas

  • Synergy of BERT and RNNs: The paper demonstrates that combining BERT's global contextual understanding with the explicit sequential modeling of BiLSTM is more effective than using either alone for log anomaly detection. This is particularly important for anomalies that depend on event order.
  • Dynamic Thresholding is Crucial: The DTPM's ability to adapt thresholds in real-time is a significant advantage. It shows that static thresholds are a major bottleneck in real-world, dynamic environments.
  • Self-Supervised Learning for Robustness: Using MLKP and VHM allows the model to learn robust representations from unlabeled data, making it practical for scenarios where labeled anomaly data is scarce.
  • Order-Sensitive Anomalies: The GR-BiLSTM module specifically addresses anomalies that are defined by the sequence of events, which is a common but challenging type of anomaly.

Key Results

  • State-of-the-Art Performance: BERT-LogAnom consistently outperformed several strong baseline methods (PCA, DeepLog, LogAnomaly, LogBERT, LAnoBERT, LogGPT, LogFiT) across all three datasets in terms of Precision, Recall, and F1-score.
  • High F1-Scores: Achieved F1-scores of 89.98% on HDFS, 89.84% on BGL, and 94.39% on Thunderbird.
  • Reduced False Positives: Achieved the highest precision (96.17%) on Thunderbird, indicating fewer false alarms.
  • Robustness to Concept Drift: In temporal split evaluations, the DTPM variant consistently achieved higher F1-scores (e.g., 90.04% on HDFS vs. 88.41% for fixed threshold), demonstrating its effectiveness in adapting to evolving log distributions.
  • Efficiency: Despite its complexity, BERT-LogAnom achieved competitive training times and an inference time of 0.9s per log, making it suitable for real-time applications.

Limitations and Future Work

  • Computational Cost: While efficient for its complexity, BERT-based models are inherently computationally intensive. The paper notes that BERT-LogAnom's training time is comparable to other BERT-based approaches.
  • Future Work: The authors suggest extending the framework to broader system environments and more complex dependency structures in large-scale log data. They also mention exploring further improvements in the GR-BiLSTM and DTPM modules.

Summary updated

Abstract

As modern software systems continue to grow in scale and structural complexity, log anomaly detection has become an essential component of system monitoring and fault diagnosis. However, existing approaches often struggle to adequately capture sequential dependencies in log data and to remain robust under distributional changes. To mitigate these issues, this paper presents BERT-LogAnom, an unsupervised framework for log anomaly detection that combines contextual representation learning, sequential modeling, and adaptive decision mechanisms. Specifically, a BERT-based encoder is employed to learn global contextual semantics from log sequences, while a gated residual bidirectional Long Short-Term Memory (GR-BiLSTM) network is introduced to model bidirectional temporal dependencies without disrupting the learned contextual information. To characterize normal system behavior from unlabeled logs, two self-supervised objectives—masked log key prediction and volume hypersphere minimization—are jointly optimized during training. Furthermore, a Dynamic Thresholding Prediction Module (DTPM) is incorporated to adjust anomaly decision boundaries in response to short-term statistical fluctuations and longer-term distribution drift. Experiments conducted on three public benchmark datasets (HDFS, BGL, and Thunderbird) show that BERT-LogAnom achieves consistently superior performance compared with representative baseline methods across precision, recall, and F1-score. Additional ablation studies further confirm the contribution of each major component in the proposed framework.
BibTeX
@inproceedings{lu2026bert,
  author = {Xi Ning Lu and Shufan An and JINGMEI CHEN and Zhan Shu and Weiping Wang and Runyi Qi and Yapeng Diao},
  title = {{BERT-LogAnom: Enhancing Log Anomaly Detection with Gated Residual BiLSTM and Dynamic Thresholding}},
  booktitle = {Electronics},
  year = {2026},
  doi = {10.3390/electronics15040806},
}
Figure