Big Data Driven Anomaly Detection for Cellular Networks

Qiqi Zhu, Li SunIEEE Access 2020 · DOI

Abstract

Anomaly detection for large scale cellular networks can be used by network operators to optimize network performance and enhance mobile user experience. This paper aims at detecting user anomalies from spatio-temporal cell phone activity data. We design an approach combining time series analysis and machine learning to extract the traffic patterns of areal units. This approach can cluster areal units with similar traffic patterns and segment a city into distinct groups. Then, in grouped-areas, we use a clustering technique to detect anomalous behaviors of the cellular network and verify the accuracy of the results using ground truth information collected from online sources. The results indicate that anomalies are associated with abruptly high or unexpected traffic demand at a specific location and time. In addition, we obtain anomaly-free data by removing anomalous data and train a decomposed traffic prediction model. It is observed that the prediction model trained with anomaly-free data can achieve lower normalized mean square error (NMSE), i.e., higher prediction accuracy, than the model trained with anomalous data.

Introduction

In modern telecommunications, network operators must identify anomalies—unexpected surges or drops in traffic—to optimize resources and prevent service outages. However, cellular traffic is highly context-dependent. A traffic volume that is standard for a bustling commercial district might represent a critical anomaly in a quiet residential suburb. Traditional anomaly detection methods often treat an entire city as a single entity, failing to account for these unique spatio-temporal rhythms. This "one-size-fits-all" approach leads to missed detections in lower-traffic areas and false alarms in high-traffic zones.

Methodology

The authors propose a "traffic-pattern-based" approach that segments the city into areas with similar behaviors before performing anomaly detection.

1. Traffic Pattern Identification

The city is divided into a grid of "areal units." For each unit, the authors construct a high-dimensional feature vector representing hourly traffic over four weeks. Using K-means clustering and the Elbow method, these units are grouped into distinct categories based on their temporal "signatures."

Temporal traffic signatures of different areal units over a seven-day period.

2. Anomaly Detection

Within each of the identified groups, K-means clustering is applied again. By using the Davies-Bouldin Index (DBI) to determine the optimal number of clusters, the system identifies "minority" clusters—those with very few data points and high centroids—as anomalies. This ensures that anomalies are detected relative to the normal behavior of similar neighborhoods.

3. Decomposed Traffic Prediction

To validate the utility of their detection method, the authors built a prediction model that decomposes traffic into a constant trend, a seasonal component (modeled via Fourier series), and a random residual. They compared the performance of this model when trained on raw data versus "anomaly-free" data.

Dataset and Validation

The study utilizes a real-world dataset from the Telecom Italia Mobile Big Data Challenge, focusing on a 100x100 grid (550 $km^2$) in Milan, Italy.

Grid-based segmentation of Milan for localized traffic analysis.

To verify the detected anomalies, the authors cross-referenced their results with external ground truth sources, such as the Municipality of Milan’s Open Data Website, to identify events like soccer matches at San Siro stadium or high-density commercial activity.

Key Results

  • Context-Aware Segmentation: The algorithm successfully segmented Milan into five groups, ranging from high-density commercial centers (0.44% of the city) to quiet residential/suburban areas (67.37%).
  • Ground Truth Validation: The system correctly identified anomalies corresponding to major public events and high-traffic periods at specific landmarks like Universita Bocconi.
  • Prediction Accuracy: The proposed decomposed prediction model achieved a 38% improvement in accuracy over the standard ARIMA benchmark.
  • Data Cleaning Impact: Training the prediction model on data cleaned by the proposed method resulted in a Normalized Mean Square Error (NMSE) of 0.1403, compared to 0.1681 for raw anomalous data.

Conclusion

The primary takeaway is that anomaly detection in cellular networks must be context-aware. By grouping geographical areas with similar traffic signatures, the sensitivity of detection is significantly increased. Furthermore, the study demonstrates that anomalies act as noise for machine learning models; removing them to create an "anomaly-free" dataset is essential for building reliable traffic forecasting tools. Future research could explore dynamic grouping to account for seasonal urban shifts and more sophisticated imputation methods for data cleaning.

Summary updated

BibTeX
@article{zhu2020big,
  author = {Qiqi Zhu and Li Sun},
  title = {{Big Data Driven Anomaly Detection for Cellular Networks}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2973214},
}
Figure