Microservice Fault Detection Method Based on BiLSTM-VAE with Multidimensional Feature Fusion

Peng Zhang, Li Weigang, Hao He, Chu YunTao, Zhan XinYue, Junsheng WuResearch Square 2025 · DOI

TL;DR

This paper introduces an unsupervised fault detection method for microservices that shifts the focus from broad service-level metrics to granular instance-level analysis. By fusing call path structures with resource utilization data using a BiLSTM-VAE model enhanced by RealNVP, the authors achieved state-of-the-art F1 scores (up to 0.985) without requiring expensive labeled training data.

What Problem Does This Paper Address?

As systems migrate to microservice architectures, they become increasingly "elastic"—constantly expanding and contracting based on load. While this is great for scalability, it’s a nightmare for fault detection.

The authors identify three core gaps in current methods:

  1. Insufficient Granularity: Most tools look at the "Service" level. However, a single failing instance (like one buggy container among ten) might be masked by load balancers, making the overall service look healthy while latency quietly creeps up.
  2. Supervised Learning Dependency: Many models require labeled "faulty" vs. "normal" data. In dynamic environments, labeling is expensive, and models trained on yesterday's architecture often fail on today's.
  3. Underutilized Data: Existing tools often look at either call chains (who calls whom) or resource metrics (CPU/RAM), but rarely both in a unified way.

Methods

The researchers propose a pipeline called BiLSTM-VAE with Multidimensional Feature Fusion. Here’s the breakdown of their "secret sauce":

  • Multidimensional Feature Trace (MFT): Instead of just looking at a list of calls, they represent the call chain as a COO (Coordinate format) matrix. This matrix fuses the structural path with "telemetry" data: response times, CPU usage, and memory utilization.
  • Edit-Distance K-means Clustering: Not all call chains are the same (e.g., a "Login" path looks different from a "Checkout" path). They use edit distance—a way to measure how different two sequences are—to cluster similar patterns before analysis.
  • BiLSTM-VAE: The core engine.
  • BiLSTM (Bidirectional Long Short-Term Memory): Captures how faults propagate both forward (errors flowing downstream) and backward (latency flowing upstream).
  • VAE (Variational Autoencoder): An unsupervised model that learns what "normal" looks like. If a new trace has a high "reconstruction error" (meaning the model can't explain it based on what it's seen), it’s likely a fault.
  • RealNVP (Real Non-Volume Preserving flows): Standard VAEs assume data follows a simple Bell curve (Gaussian). Real-world microservice data is messy, with "long tails" and multiple peaks. RealNVP helps the model map these complex distributions more accurately.
  • P-Value Hypothesis Testing: The final step uses Kernel Density Estimation (KDE) to determine the statistical probability that a trace is an anomaly, allowing for real-time fault determination.

Datasets

The authors validated their approach on two major benchmarks:

  1. 2020AIOps International Challenge Dataset: Real-world production data from Zhejiang Mobile, containing 25 services, 116 instances, and over 175,000 call chains.
  2. TT (TrainTicket) Dataset: A popular open-source microservice benchmark used to evaluate failure detection systems, featuring 41 microservices.

Key Insights and Ideas

  • The "Masking" Effect: The paper highlights how load balancers are the enemy of fault detection. By averaging performance across instances, they hide the "limping" node. Moving to instance-level detection is the only way to find these "gray failures."
  • Bidirectional Propagation is Critical: Faults aren't one-way streets. A disk overflow in a database (downstream) causes a "fail-retry" loop that increases latency in the API gateway (upstream). The BiLSTM architecture is specifically chosen to catch this "reverse" propagation.
  • Unsupervised is the Future: By learning the "baseline" of normal operations, the system adapts to version iterations and architectural changes without needing a human to label new faults.

Key Results

The proposed method outperformed several state-of-the-art baselines (like TraceGra and STS-AD):

  • F1 Scores: Achieved 0.979 on the 2020AIOps dataset and 0.985 on the TT dataset.
  • Improvement: This represents a 1.3% and 0.9% increase over the best existing baseline methods.
  • Ablation Success: Removing any single component (the multidimensional fusion, the clustering, or the RealNVP module) led to a significant drop in performance, proving that the integration of these different techniques is what drives the accuracy.

Limitations and Future Work

While the results are impressive, the authors note some areas for growth:

  • Hyperparameter Sensitivity: The performance depends on choosing the right number of clusters ($K$) and hidden units. If $K$ is too small, different patterns interfere; if too large, the model overfits.
  • Computational Overhead: While the COO matrix is compact, running BiLSTM-VAEs in real-time for massive systems requires significant compute.
  • Future Directions: The authors suggest that future work could involve more automated ways to select these hyperparameters based on the specific characteristics of the microservice call chains.

Summary updated

Abstract

<title>Abstract</title> The complex dynamic dependency and elastic expansion and contraction mechanisms of microservice architectures lead to system faults with hidden propagation paths and diverse anomaly patterns. Traditional fault detection methods have limitations in terms of detection granularity, dependency on supervised learning, and utilization of multi-source data. This paper proposes a multidimensional feature fusion fault detection method based on BiLSTM-VAE. The multidimensional feature call chain (MFT) is constructed by a coordinate format matrix, fusing multi-source data such as instance-level call path structure, response time and resource utilisation; the call chain pattern is classified by edit-distance K-means clustering, combining with BiLSTM-VAE model to capture bidirectional spatial-temporal features of the call chain, and introducing RealNVP a posteriori flow to optimise the modeling of the hidden variable distribution; based on the kernel density estimation The baseline distribution of normal call chain is established, and p-value hypothesis testing is utilised to realize real-time fault determination. Experiments demonstrate that the method achieves F1 values of 0.979 and 0.985 on the 2020AIOps and TT datasets, respectively, which are 1.3% and 0.9% higher than the optimal baseline. The ablation experiments validate the effectiveness of multidimensional feature fusion, pattern clustering, and a posteriori flow modules. The method breaks through supervisory dependence through unsupervised learning, significantly improving the accuracy and robustness of instance-level fault detection.
BibTeX
@inproceedings{zhang2025microservice,
  author = {Peng Zhang and Li Weigang and Hao He and Chu YunTao and Zhan XinYue and Junsheng Wu},
  title = {{Microservice Fault Detection Method Based on BiLSTM-VAE with Multidimensional Feature Fusion}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-7508236/v1},
}
Figure