Forecasting Loss of Signal in Optical Networks with Machine Learning

Wenjie Du, David Côté, Chris Barber, Yan Liu

Abstract

Optical network reliability is critical for global communications, yet "Loss of Signal" (LOS) events often cause significant disruptions. This paper presents a machine learning framework developed by Concordia University and Ciena to forecast LOS events 1 to 7 days in advance. By utilizing a specialized deep learning architecture (BRITS) and a "mega-dataset" spanning six international networks, the authors demonstrate that proactive failure prediction is possible even with sparse, unlabeled industrial data featuring missing value rates over 80%.

Introduction

Optical networks are the backbone of modern communication, but they are susceptible to various failures ranging from physical fiber cuts to equipment degradation. While sudden cuts are inherently unpredictable, many failures are preceded by subtle performance shifts—"digital groans"—in telemetry data. However, predicting these events is difficult due to three primary factors: massive missing data (often exceeding 70%), a lack of explicit failure labels in raw logs, and the challenge of generalizing models across diverse network configurations and hardware.

Methodology

The researchers implemented a multi-stage pipeline to address these challenges. First, they employed a rule-based labeling system to overcome the lack of ground-truth labels. By analyzing metrics such as Unavailable Seconds (UAS) and High Correction Count Seconds (HCCS), they retrospectively identified failure states and labeled the preceding 7 days as "pre-failure" samples within a 14-day sliding window.

Data preprocessing and rule-based labeling strategy using a 14-day sliding window.

To handle the extreme data sparsity, the team utilized the Bidirectional Recurrent Imputation for Time Series (BRITS) architecture. Unlike standard imputation methods that use simple means or medians, BRITS treats missing values as learnable variables within an RNN framework. The model simultaneously performs data imputation and failure classification, ensuring the forecasting process is robust to the high missing data rates found in industrial settings.

The BRITS-based architecture combining an imputer for missing data and a classifier for LOS forecasting.

Furthermore, a transfer learning approach was adopted. A "mega-dataset" was created by aggregating data from six global networks to pre-train a model on general failure signatures, which was then fine-tuned for specific local environments to combat the "needle-in-a-haystack" problem of rare failures.

Datasets

The study leveraged a massive corpus of real-world Performance Monitoring (PM) data covering 1,373 days across six international networks. This included approximately 16.5 million samples from 94,000 ports, encompassing 12 facility types and 9 protocols. The aggregated "mega-dataset" featured 125 distinct features but suffered from an average missing data rate of 82.2%.

Results and Discussion

The results confirm that "Imminent LOS" (ILOS) is a forecastable phenomenon. The authors observed that signal quality (Q-factor) often degrades incrementally over months before a total outage. Key findings include:

  • Forecasting Window: LOS events were successfully predicted 1-7 days in advance.
  • Model Performance: The BRITS-based model achieved 65% precision at 1% recall across all facility types, outperforming Random Forest baselines by approximately 39%. For high-priority 100G OTN lines, precision reached over 80%.
  • Transfer Learning: The "mega-dataset" approach improved performance on smaller, data-poor networks by up to 75.8% compared to standard XGBoost baselines.
  • Operational Utility: By prioritizing precision over recall, the system minimizes "alert fatigue," providing operators with a small number of highly reliable alarms rather than a flood of noisy ones.

Conclusion

This research demonstrates that deep learning can effectively forecast optical network failures despite significant data quality issues. While sudden physical interruptions remain unpredictable, the "slow fade" of equipment can be caught days in advance. Future work will focus on multi-class prediction to identify specific root causes and utilizing higher-resolution data to capture transient performance issues that daily-binned data might miss.

Summary updated

BibTeX
@article{du2021forecasting,
  author = {Wenjie Du and David Côté and Chris Barber and Yan Liu},
  title = {{Forecasting Loss of Signal in Optical Networks with Machine Learning}},
  journal = {JOCN},
  year = {2021},
  doi = {10.1364/jocn.423667},
}
Figure