Abstract
Network and Service analytics can harness the immense stream of operational data from clouds, to services, to social and communication networks. In the era of big data and connected devices of all varieties, analytics and machine learning have found ways to improve reliability, configuration, performance, fault and security management. In particular, we see a growing trend towards using machine learning, artificial intelligence and data analytics to improve operations and management of information technology services, systems and networks.
Introduction
Modern networks—ranging from massive cloud data centers to 5G mobile infrastructure—generate an immense stream of operational data. Historically, managing these systems relied on manual rules and human operators. However, the sheer scale of "big data" and the variety of connected devices have made traditional management unsustainable.
This special section addresses the critical need for automated, data-driven decision-making. The core challenge lies in making machine learning (ML) algorithms reliable, scalable, and "dependable" enough to handle real-world tasks like fault management, security, and performance optimization without constant human intervention.
Methodologies in Automated Management
This editorial introduces a curated collection of 15 research papers that leverage ML and data analytics to automate the management of complex modern networks. Key technical approaches highlighted include:
- Deep Reinforcement Learning (DRL): Employed for defending against botnet evasion attacks and optimizing mobility management in mobile networks.
- Hierarchical Transformers: The "HitAnomaly" structure captures semantic information in system logs to enhance anomaly detection.
- Graph Convolutional Networks (GCNs): Applied to social media data to detect rumor propagation by analyzing structural connections between users.
- Few-Shot Learning: Used to characterize the state of LTE base stations even when labeled training data is scarce.
- Recommender Systems: Utilized to model dynamics between different Self-Organizing Network (SON) functions, preventing conflicts in cellular management.
Diverse Datasets and Real-World Applications
The featured research utilizes a wide range of real-world and simulated datasets to validate these new approaches:
- Network Traffic: Includes captures from hundreds of routers (Flowyager) and encrypted HTTP adaptive video streaming data.
- Cloud Infrastructure: Apache Spark workloads, Docker-based containerized logs, and general system maintenance logs.
- Mobile & Radio Networks: Live Long-Term Evolution (LTE) network data, radio performance statistics, and urban radio transmitter location data.
- Social Media Analytics: Approximately one million COVID-19-related tweets, data from the 2019 Spanish General Election, and the Sina Weibo dataset.
Key Insights
Several standout insights emerge from this collection:
- QoE Inference from Encrypted Traffic: Systems like ViCrypt can monitor Quality of Experience (QoE)—such as video stalling or resolution—by using ML to infer performance from traffic patterns rather than packet content.
- Efficiency in Auto-Tuning: The "You Only Run Once" approach for Apache Spark demonstrates that ML can predict optimal configurations from a single workload run, significantly reducing testing overhead.
- Social-Network-as-a-Sensor: There is a growing trend of treating social media as a network management problem, using analytics to identify "political bots" and misinformation (infodemics) that disrupt crisis management.
- Proactive Mobility: By combining Long Short-Term Memory (LSTM) networks with Reinforcement Learning, researchers can predict group user trajectories, allowing networks to "pre-migrate" services before a user moves to a new cell tower.
Significant Results
The editorial highlights several quantitative and qualitative milestones:
- Scalability: Flowyager enables querying network-wide flow data across hundreds of routers simultaneously.
- Resource Optimization: New ML-based CPU allocation methods for containers reduce job completion times compared to standard static allocation.
- Accuracy: ML models for radio coverage were successfully benchmarked to optimize transmitter placement in complex urban environments.
- Selectivity: Out of 83 submissions, only 15 were chosen for Part I, representing a highly competitive 18% acceptance rate.
Conclusion and Future Outlook
While these papers show great promise, several hurdles remain. There is a continuing need to ensure ML algorithms can scale to global networks without becoming "brittle" or failing under unexpected conditions. Future research should focus on defining "novel platforms" that can natively harness vast operational data to drive automated decisions in real-time. This editorial covers the first half of the special section; further research is slated for publication in Part II (March 2021).
Summary updated