TL;DR
This paper introduces a comprehensive framework for evaluating synthetic network traffic, revealing that Bayesian Networks (BNs) outperform Generative Adversarial Networks (GANs) in generating realistic, diverse, and protocol-compliant network flows, while also being more computationally efficient. This work highlights the limitations of current GAN approaches for tabular data and advocates for BNs as a superior solution for synthetic network traffic generation.
What Problem Does This Paper Address?
Evaluating network security systems, like intrusion detection systems (NIDS), critically depends on high-quality, labeled network traffic data. However, obtaining such data is challenging due to privacy concerns, the labor-intensive nature of labeling, and the complexity and cost of simulating realistic network environments. Synthetic data generation offers a promising solution, but existing methods, particularly GANs, struggle to accurately capture the intricate statistical properties and protocol adherence of real network traffic. This paper addresses the gap by proposing a robust evaluation framework and demonstrating that Bayesian Networks can generate superior synthetic network traffic compared to state-of-the-art GANs.
Methods
The authors propose a novel evaluation framework for synthetic network traffic that goes beyond traditional metrics. It assesses four key criteria: Realism (how closely generated data matches real data distributions), Diversity (the variance and spread of generated data), Novelty (ensuring generated data isn't just copied from training data), and Compliance (adherence to network protocol rules).
For evaluation, they employ:
- Marginal Distribution Comparison: Using Jensen-Shannon Divergence (JSD) for categorical features and Earth Mover's Distance (EMD) for numerical features.
- Conditional Probability Comparison: Using Pairwise Conditional Distribution (PCD) for numerical features and Contingency Matrix Differences (CMD) for categorical features to assess inter-feature correlations.
- Joint Distribution Comparison: Employing Density and Coverage metrics to evaluate realism and diversity at the sample level.
- Novelty Assessment: Using Membership Disclosure (MD) to detect copied samples.
- Compliance Evaluation: Adapting the Domain Knowledge Check (DKC) to verify adherence to network protocol rules.
The core generative method proposed is Bayesian Networks (BNs). They explore two BN-specific strategies:
- BNbins: A discretization strategy that groups numerical values into quantiles.
- BNGM: A discretization strategy using Variational Gaussian Mixture Models (VGMM).
They also compare these BN approaches against several GAN-based methods (E-WGAN-GP, NetShare, CTGAN) and a naive sampling method.
Datasets
The experiments were conducted on three real-world network traffic datasets:
- CICSmallFeatureSet: A subset of CIC-IDS2017 with 11 features.
- CICLongFeatureSet: A larger subset of CIC-IDS2017 with 30 features.
- UGR'16: A dataset from a Spanish ISP, preprocessed by NetShare authors.
Key Insights and Ideas
- GANs struggle with tabular data: While successful in image generation, GANs face challenges with the discrete and highly correlated nature of network traffic data, leading to issues like mode collapse and difficulty in capturing inter-feature dependencies.
- BNs excel at capturing dependencies: The inherent structure of Bayesian Networks allows them to explicitly model and learn conditional dependencies between network flow features, leading to more realistic and compliant synthetic data.
- Comprehensive evaluation is crucial: Relying on a single metric can be misleading. The proposed framework, encompassing Realism, Diversity, Novelty, and Compliance, provides a more holistic view of synthetic data quality.
- Protocol compliance matters: Many GAN-based methods fail to generate traffic that adheres to fundamental network protocols, a critical aspect for realistic evaluation datasets.
- Computational efficiency: BNs offer a significant advantage in terms of training time and resource usage compared to complex GAN models.
Key Results
- BNs outperform GANs: Across all three datasets and most evaluation metrics, the proposed BN-based methods (BNbins and BNGM) consistently outperformed GAN-based approaches in generating realistic and diverse network traffic.
- Superior Compliance: BN-based methods achieved significantly lower DKC scores (indicating better compliance) compared to most GANs, with NetShare and the Naive Sampler showing particularly poor compliance.
- Efficiency Gains: BNbins, in particular, demonstrated the fastest preprocessing and training times, making it a highly efficient solution for synthetic network traffic generation.
- GAN Limitations Highlighted: CTGAN showed a tendency towards "mode invention" (generating unrealistic samples), while NetShare struggled with both realism and compliance.
- Real-world data consistency: The evaluation results were consistent across different datasets, including real-world traffic from UGR'16, reinforcing the robustness of the findings.
Limitations and Future Work
- Low Cardinality Focus: The current study primarily uses datasets with relatively low cardinality for categorical features. Future work could extend the findings to datasets with higher cardinality, such as incorporating IPv6 addresses and a wider range of port values.
- Temporal Dependencies: The current BN approach does not explicitly model temporal dependencies in network traffic. The authors hypothesize that BNs could be extended to capture these dynamics, which would be a valuable direction for future research.
- NIDS Evaluation: The authors suggest using the generated synthetic NFs in an actual NIDS evaluation pipeline to assess their practical usability and effectiveness in a cybersecurity context.
Summary updated