Lumos: A Library for Diagnosing Metric Regressions in Web-Scale Applications

Jamie Pool, Ebrahim Beyrami, Vishak Gopal, Ashkan Aazami, Jayant Gupchup, Jeff Rowland, Binlong Li, Pritesh Kanani, Ross Cutler, Johannes GehrkeKDD 2020 · DOI · arXiv

Introduction

In massive web-scale services like Microsoft Teams and Skype, engineering teams monitor thousands of Key Performance Indicators (KPIs). A critical challenge in maintaining these services is distinguishing between "real" regressions caused by code changes and "false positives" driven by external environmental shifts, such as regional holidays or changes in user device demographics. Manually investigating these anomalies is a "needle in a haystack" problem that often consumes days of engineering effort.

Lumos is an open-source Python library designed to automate this diagnosis. By reframing anomaly detection as an observational A/B testing problem, Lumos identifies whether a metric shift is a true system failure or merely a byproduct of population bias.

The Lumos Diagnostic Workflow

Lumos operates through a phased workflow that applies statistical rigor to telemetry data. It treats the period before a detected regression as the "Control" group and the period during the regression as the "Treatment" group.

The Lumos diagnostic workflow for identifying and ranking metric regressions.

The process begins with data pre-processing, including automated binning and one-hot encoding for high-cardinality categorical data. The system then performs a statistical significance check on the metric delta (Δ). If significant, it moves to the core innovation: the bias check.

Addressing Population Bias

A primary source of noise in anomaly detection is "population bias"—shifts in invariant features like OS version, country, or network type that should not be affected by a software release. Lumos uses Propensity Score Matching (PSM) via a Random Forest model to determine if the observed metric change can be explained by these shifts.

Comparison of unmatched and matched population distributions, demonstrating how propensity score matching balances covariates to isolate the true effect of a regression.

If the regression disappears after normalizing for these shifts, it is flagged as a Type-B anomaly (false positive). If the regression persists, it is classified as a Type-S anomaly (true regression). For these real issues, Lumos employs a Hazard Score to rank hypothesis features. This score calculates the difference in the conditional probability of a feature's occurrence given a failure, allowing engineers to quickly identify the specific component or condition (e.g., a specific video codec) causing the regression.

Evaluation and Impact

The library was validated using real-world production data from Microsoft’s Real-Time Communication (RTC) stack, including a set of ~100 historical incidents with known ground truths.

Key results from production deployment include:

  • Alert Reduction: Lumos successfully filtered out 90% to 98% of initial alerts as false positives across metrics like Call Drops and Signaling Failures.
  • Efficiency Gains: Engineering teams reported a 65% to 95% reduction in time spent on manual metric-based investigations.
  • Interpretability: By using univariate ranking rather than complex black-box models, Lumos provides prioritized lists of features that are easily interpretable by on-call engineers.

Conclusion and Future Work

Lumos demonstrates that population bias is the primary driver of noise in web-scale anomaly detection. While the current version focuses on binary success/fail metrics and requires manual selection of invariant features, future iterations aim to support continuous metrics like latency and introduce automated feature clustering to handle multicollinearity. By combining domain knowledge with automated statistical matching, Lumos transforms a complex manual task into a streamlined, automated diagnostic process.

Summary updated

BibTeX
@inproceedings{pool2020lumos,
  author = {Jamie Pool and Ebrahim Beyrami and Vishak Gopal and Ashkan Aazami and Jayant Gupchup and Jeff Rowland and Binlong Li and Pritesh Kanani and Ross Cutler and Johannes Gehrke},
  title = {{Lumos: A Library for Diagnosing Metric Regressions in Web-Scale Applications}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2020},
  doi = {10.1145/3394486.3403306},
}
Figure