Introduction
In massive web-scale services like Microsoft Teams and Skype, engineering teams monitor thousands of Key Performance Indicators (KPIs). A critical challenge in maintaining these services is distinguishing between "real" regressions caused by code changes and "false positives" driven by external environmental shifts, such as regional holidays or changes in user device demographics. Manually investigating these anomalies is a "needle in a haystack" problem that often consumes days of engineering effort.
Lumos is an open-source Python library designed to automate this diagnosis. By reframing anomaly detection as an observational A/B testing problem, Lumos identifies whether a metric shift is a true system failure or merely a byproduct of population bias.
The Lumos Diagnostic Workflow
Lumos operates through a phased workflow that applies statistical rigor to telemetry data. It treats the period before a detected regression as the "Control" group and the period during the regression as the "Treatment" group.
The process begins with data pre-processing, including automated binning and one-hot encoding for high-cardinality categorical data. The system then performs a statistical significance check on the metric delta (Δ). If significant, it moves to the core innovation: the bias check.
Addressing Population Bias
A primary source of noise in anomaly detection is "population bias"—shifts in invariant features like OS version, country, or network type that should not be affected by a software release. Lumos uses Propensity Score Matching (PSM) via a Random Forest model to determine if the observed metric change can be explained by these shifts.
If the regression disappears after normalizing for these shifts, it is flagged as a Type-B anomaly (false positive). If the regression persists, it is classified as a Type-S anomaly (true regression). For these real issues, Lumos employs a Hazard Score to rank hypothesis features. This score calculates the difference in the conditional probability of a feature's occurrence given a failure, allowing engineers to quickly identify the specific component or condition (e.g., a specific video codec) causing the regression.
Evaluation and Impact
The library was validated using real-world production data from Microsoft’s Real-Time Communication (RTC) stack, including a set of ~100 historical incidents with known ground truths.
Key results from production deployment include:
- Alert Reduction: Lumos successfully filtered out 90% to 98% of initial alerts as false positives across metrics like Call Drops and Signaling Failures.
- Efficiency Gains: Engineering teams reported a 65% to 95% reduction in time spent on manual metric-based investigations.
- Interpretability: By using univariate ranking rather than complex black-box models, Lumos provides prioritized lists of features that are easily interpretable by on-call engineers.
Conclusion and Future Work
Lumos demonstrates that population bias is the primary driver of noise in web-scale anomaly detection. While the current version focuses on binary success/fail metrics and requires manual selection of invariant features, future iterations aim to support continuous metrics like latency and introduce automated feature clustering to handle multicollinearity. By combining domain knowledge with automated statistical matching, Lumos transforms a complex manual task into a streamlined, automated diagnostic process.
Summary updated