Always be Pre-Training: Representation Learning for Network Intrusion Detection with GNNs

Zhengyao Gu, Diego Troy Lopez, Lilas Alrahis, Ozgur Sinanoglu

TL;DR

Researchers from NYU have developed a new approach for Network Intrusion Detection Systems (NIDS) that combines Graph Neural Networks (GNNs) with self-supervised "in-context" pre-training. By moving away from label-heavy pre-processing and leveraging unlabeled network traffic, their model achieves over 98% of state-of-the-art performance using less than 4% of the labeled data typically required.

What Problem Does This Paper Address?

Modern Network Intrusion Detection Systems (NIDS) are increasingly turning to Machine Learning, specifically Graph Neural Networks (GNNs), because network traffic is inherently relational. GNNs can "see" the topology of an attack—like a coordinated DDoS or a Man-in-the-Middle sequence—rather than just looking at isolated packets.

However, there is a major bottleneck: label dependency. Most high-performing GNN-based NIDS rely on "target encoding" for categorical features (like protocol types or flags). Target encoding requires knowing the labels (benign vs. malicious) during the data pre-processing stage to calculate correlations. In the real world, labeling network traffic is incredibly expensive and time-consuming. This creates a "chicken and egg" problem where you need a labeled dataset from your specific network environment before you can even build the detector for that environment.

Methods

The authors propose two key shifts to break this dependency:

  1. Dense Vector Representations: Instead of using scalar-valued target encodings that compress categorical data into a single number based on labels, they use dense embeddings. They treat categorical features as high-dimensional vectors (via one-hot encoding followed by a linear projection). This preserves the rich semantic information of network protocols without needing labels upfront.
  2. In-Context Pre-training: The team introduces a two-stage training pipeline.
  3. Stage 1 (Self-Supervised): They use an encoder (E-GraphSAGE) and a discriminator in a setup called Anomal-E (based on Deep Graph Infomax). The model looks at vast amounts of unlabeled traffic from the target network and learns to distinguish between real network structures and "corrupted" ones (where features are randomly shuffled). This "in-context" pre-training allows the model to learn the "baseline" behavior of that specific network.
  4. Stage 2 (Fine-Tuning): The pre-trained encoder is then connected to a classification head and fine-tuned on a very small sample of labeled data.

Datasets

The study evaluated the approach on two prominent heterogeneous NIDS benchmarks:

  • ToN-IoT: A dataset featuring 40 features across various protocols (TCP, UDP, HTTP, DNS) with 9 different attack types. It is known for having a high proportion of categorical features (31 out of 40).
  • NF-UQ-NIDS-V2: A massive, standardized NetFlow dataset compiled from four different sources, containing over 75 million records and 20 attack categories. This represents a much more complex and challenging "real-world" simulation.

Key Insights and Ideas

  • SSL as a Weight Initializer: The "aha moment" here is that Self-Supervised Learning (SSL) isn't just for unsupervised anomaly detection. It serves as a superior initialization for supervised tasks. By learning the structure of the network first, the GNN starts the supervised phase with a "gut feeling" for what normal traffic looks like.
  • Dense > Scalar: The researchers found that while target encoding is a common shortcut, it loses vital information. Dense embeddings capture the nuances of protocol behavior that are essential for detecting "zero-day" or novel attacks that haven't been correlated with labels yet.
  • The Power of "In-Context": Unlike computer vision, where you might pre-train on ImageNet and fine-tune on medical X-rays, NIDS benefits most from pre-training on the same network it will eventually monitor. This "in-context" approach ensures the model understands the specific quirks of the local environment.

Key Results

The results demonstrate a massive leap in data efficiency:

  • Extreme Few-Shot Performance: On the NF-UQ-NIDS-V2 dataset, the pre-trained model (D-SSL) achieved an F1-score of 84.64% using only 3.7% of the labeled training data. This is 95.11% of the performance reached when using the entire labeled dataset.
  • Stability: Pre-training significantly reduced the variance in model performance. On ToN-IoT, it decreased variance by over 82%, making the detector much more reliable across different training runs.
  • Outperforming Baselines: The dense SSL approach consistently outperformed standard GNNs and traditional Machine Learning models like Decision Trees, especially as the complexity of the dataset increased.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Graph Complexity: The current study uses a relatively basic GNN backbone (E-GraphSAGE). Exploring more advanced architectures like Graph Transformers could yield even higher accuracy.
  • Directionality: The current model treats network flows as undirected edges. In reality, the "who started the conversation" aspect (source vs. destination) is critical for security, and incorporating edge directionality is a planned next step.
  • Real-World Deployment: The study relies on synthetic benchmarks. The ultimate test will be deploying this "Always be Pre-Training" workflow on live, production-grade network traffic to see how it handles the noise of a non-simulated environment.

Summary updated

Abstract

Graph neural network-based network intrusion detection systems have recently demonstrated state-of-the-art performance on benchmark datasets. Nevertheless, these methods suffer from a reliance on target encoding for data pre-processing, limiting widespread adoption due to the associated need for annotated labels—a cost-prohibitive requirement. In this work, we propose a solution involving in-context pre-training and the utilization of dense representations for categorical features to jointly overcome the label-dependency limitation. Our approach exhibits remarkable data efficiency, achieving over 98% of the performance of the supervised state-of-the-art with less than 4% labeled data on the NF-UQ-NIDS-V2 dataset.
BibTeX
@inproceedings{gu2024always,
  author = {Zhengyao Gu and Diego Troy Lopez and Lilas Alrahis and Ozgur Sinanoglu},
  title = {{Always be Pre-Training: Representation Learning for Network Intrusion Detection with GNNs}},
  booktitle = {ISQED},
  year = {2024},
  url = {https://openreview.net/forum?id=mreKegbGrV},
}

Loading related content...

Figure