Overcoming Network Data Scarcity for Machine Learning

18 concepts14 papers1 proposal

Every second, global internet infrastructure routes exabytes of traffic. Yet, researchers trying to build machine learning (ML) models for network optimization, security, and automation face a chronic, severe shortage of usable data. This paradox defines the modern landscape of data-driven networking. While computer vision and natural language processing have thrived on massive, open-access datasets, networking research remains constrained by a bottleneck known as network data scarcity.

The Privacy and Proprietary Deadlock

The scarcity of network data is not due to a lack of raw traffic, but rather a combination of strict privacy regulations, commercial sensitivities, and structural barriers.

As Mudhakar Srivatsa et al. point out in their work on foundation models for networking, network headers and payloads are heavily guarded by privacy laws. Sharing production-grade network logs risks exposing sensitive user information or proprietary enterprise infrastructure. Consequently, telecommunications operators and enterprises lock their operational data within proprietary silos, as highlighted by Yan Xin et al. in the context of 5G and 6G automation.

Even when raw data can be collected internally, labeling it accurately requires immense human effort and deep domain expertise. In cellular wireless networks, for instance, Tao Zhang et al. note that the inherent complexity of data collection infrastructure makes acquiring comprehensive, labeled datasets for self-healing systems exceptionally difficult (Data-Driven Machine Learning Techniques for Self-Healing in Cellular Wireless Networks).

The Cost of Fragmentation

Without centralized, high-quality benchmarks, the networking field suffers from severe fragmentation (Haitham Afifi et al. in Machine Learning With Computer Networks). Researchers are frequently forced to evaluate their systems using internal, heavily anonymized, or localized datasets (Fabian Fischer et al. in Large-Scale Network Monitoring for Visual Analysis of Attacks).

This lack of standardization makes it nearly impossible to objectively compare different ML models or guarantee that a solution will generalize to a different network architecture. This issue spans multiple domains, including Network Function Virtualization (NFV), where the scarcity of shared data impedes reproducible research and algorithm comparison (Houda Jmila et al. in The Promise of Applying Machine Learning Techniques to Network Function Virtualization), as well as general ML-driven networking systems (Xiaowen Chu et al. in Guest Editorial: Interplay between Machine Learning and Networking Systems).

This data gap is particularly dangerous in network security and anti-censorship:

The Solution Space: Synthesizing and Pre-training

To break this deadlock, researchers are shifting away from the traditional paradigm of collecting and manually labeling raw production traffic. Instead, the community is exploring three core strategies: Foundation Models, Generative Data Augmentation, and Network Digital Twins.

1. Foundation Models for Unlabeled Data

Instead of relying on scarce human-labeled datasets, foundation models exploit the massive volume of unlabeled data already traversing the wires. Mudhakar Srivatsa et al. propose adapting architectures like BERT or GPT to networking (Rethinking Data-driven Networking with Foundation Models). By pre-training models on self-supervised tasks using raw, unlabeled network traces, the models learn the underlying structural grammar of network traffic. This drastically reduces the amount of expensive, labeled data needed for downstream tasks like traffic classification or anomaly detection.

2. Protocol-Constrained Generative AI

When real data cannot be shared due to privacy constraints, generative models can step in to synthesize high-fidelity alternatives. Yinqiu Liu et al. highlight Deep Generative Models (DGMs) as a key tool for simulating network environments during idle periods or new deployments where historical logs are missing (Deep Generative Model and Its Applications in Efficient Wireless Network Management).

However, standard generative models (like vanilla GANs) often produce traffic that violates strict networking protocols, making the data useless for real-world tools. To solve this, Xi Jiang 0007 et al. developed NetDiffusion (NetDiffusion: Network Data Augmentation Through Protocol-Constrained Traffic Generation). This framework maps packet headers into image-like representations and applies diffusion models constrained by protocol heuristics. This ensures the generated synthetic traffic is not only statistically accurate but also fully compatible with standard network analysis tools like Wireshark.

[Raw Packet Headers] ──> [Image-like Representation] ──> [Diffusion Model + Protocol Constraints] ──> [Synthesized Valid Traffic]

3. Digital Twins and Open Challenges

Another powerful approach is the creation of network "Digital Twins"—high-fidelity virtual replicas of physical networks. Yongming Huang et al. demonstrate how generative AI can power these digital twins to simulate rare edge-case scenarios that are too costly or dangerous to replicate in production (Wireless Network Digital Twin for 6G: Generative AI as A Key Enabler). By moving performance modeling from slow, rigid simulations to agile, data-driven digital twins, researchers can execute complex "what-if" analyses in real time (Linbo Hui et al. in Digital Twin for Networking).

Finally, to democratize access to what little data exists, community-driven initiatives are vital. Competitions like the Graph Neural Networking Challenge organized by José Suárez-Varela et al. provide curated, high-quality datasets to the broader research community (The graph neural networking challenge), bridging the educational and resource gap for training the next generation of network engineers.

Key Foundational Work

Large-Scale Network Monitoring for Visual Analysis of Attacks (2008) — contributed 1 concept elements to this direction.
Network Traffic Obfuscation and Automated Internet Censorship (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_network_data_scarcity_3355b50a,
  title = {network data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/network-data-scarcity/rdr_3355b50a5ccf43d8a5517c567a0dbc87},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_network_data_scarcity_3355b50a_v1,
  title = {network data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/network-data-scarcity/rdr_3355b50a5ccf43d8a5517c567a0dbc87/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake