Leakage-free evaluation of residual graph convolutional networks for network intrusion detection

Zahid IqbalarXiv 2026 · arXiv

TL;DR

This paper reveals that the high performance of Graph Neural Networks (GNNs) in network intrusion detection is often inflated by data leakage. When evaluated under a strict, "leakage-free" protocol, GNNs actually fail to outperform simple multi-layer perceptrons or traditional tree ensembles in accuracy, though they demonstrate a remarkable and unique robustness to feature noise.

What Problem Does This Paper Address?

In the world of Network Intrusion Detection Systems (NIDS), researchers are increasingly turning to Graph Neural Networks. The idea is intuitive: network traffic isn't just a series of independent events; it’s a web of connections. However, this paper argues that many "state-of-the-art" GNN results are built on shaky ground due to data leakage.

The authors identify three main sources of "optimism" in current literature:

  1. Transductive Graph Construction: Including test data features in the graph during the training phase.
  2. Data Duplication: Allowing identical traffic flows to appear in both training and testing sets.
  3. Single-Split Reporting: Relying on a single random split of data, which can hide the variance in model performance.

The goal of this study is to establish a rigorous, leakage-controlled baseline to see what a Residual Graph Convolutional Network (R-GCN) actually contributes once these shortcuts are removed.

Methods

The authors propose an inductive evaluation protocol designed to be as realistic as possible.

Key components include:

  • Label-Aware De-duplication: Removing exact duplicate flows before splitting the data. This is critical; in one dataset, 69% of the flows were duplicates.
  • Inductive kNN Graph: Instead of a global graph, they build a $k$-nearest-neighbor graph using only training flows. At inference time, new test flows are "attached" to their nearest training neighbors, ensuring the model never "sees" the test set structure during training.
The inductive graph construction ensures that test/validation flows only receive messages from training flows, preventing structural leakage.
  • Residual GCN (R-GCN): They use a two-layer GCN with identity skip connections (similar to ResNet) to prevent "over-smoothing," where node representations become indistinguishable as the graph gets deeper.
The R-GCN architecture uses 64 hidden units per layer and an identity skip connection to preserve feature information across layers.

Datasets

The study utilizes two major NIDS benchmarks:

  1. CICIDS2017: A widely used dataset containing various attack types like DoS, DDoS, and Botnets.
  2. NF-UNSW-NB15-v2: A NetFlow-based version of the UNSW-NB15 dataset.

The authors highlight the impact of cleaning: for CICIDS2017, the flow count dropped from 2.8 million to 2.5 million. For NF-UNSW-NB15-v2, it plummeted from 2.3 million to just 731,810.

Table 2. Class distribution of CICIDS2017 after cleaning.

Key Insights and Ideas

  • The "Message Passing" Myth: Surprisingly, the R-GCN performed almost identically to a simple Multi-Layer Perceptron (MLP) that ignored the graph entirely. This suggests that in these benchmarks, the "neighborhood" information provided by the graph doesn't add much discriminative power beyond what's already in the individual flow features.
  • Trees are Still Kings: Traditional tree ensembles like XGBoost and LightGBM consistently outperformed the neural models in binary and multiclass detection.
  • The Robustness Edge: The real "Aha!" moment comes from the noise stress test. While XGBoost's performance collapsed when faced with even moderate feature noise, the R-GCN remained highly stable. This suggests GNNs might be better suited for "dirty" real-world data where sensors might be imprecise or features slightly perturbed.
  • Structural Comparison: The authors provide a helpful comparison (Table 1) showing how their study differs from previous work, particularly in how they handle test flow edges and data splits.

Key Results

  • Binary Accuracy: On CICIDS2017, the R-GCN achieved a binary F1 score of 98.11%, while XGBoost reached 99.67%.
  • Multiclass Struggle: The gap widened in multiclass detection. XGBoost hit a macro-F1 of 93.08%, whereas the R-GCN managed only 64.51%. The R-GCN struggled significantly with small attack classes, often misclassifying benign traffic as rare attacks.
  • Noise Robustness: Under Gaussian noise (SD 0.5), the R-GCN maintained an F1 of 96.16%. In contrast, XGBoost’s F1 score fell to a staggering 0.67%.

Limitations and Future Work

The primary limitation identified is the GNN's poor performance on minority classes (class imbalance). The models tended to generate many false alarms for small attack families like "Bot" or "Web" attacks.

The authors suggest that future work should focus on:

  • Improving precision on small attack classes, perhaps through better loss functions or second-stage classifiers.
  • Exploring Federated Learning to train these models across different organizations without sharing raw, sensitive traffic data.
  • Optimizing these models for the limited computing and energy budgets of IoT and smart-city devices.

Summary updated

Abstract

Graph neural networks are increasingly proposed for network intrusion detection, yet many reported results come from protocols in which test flows shape the graph used for training, duplicated flows sit on both sides of the split, or a single split is reported. This study measures what a residual graph convolutional network gains over simpler models once these sources of optimism are removed. CICIDS2017 and NF-UNSW-NB15-v2 were cleaned with label-aware removal of exact duplicates, removing 69% of NF-UNSW-NB15-v2 flows. Flows were linked in a nearest-neighbour graph built from training flows only, and validation and test flows were attached to their nearest training flows at prediction time. A two-layer residual network was compared with eight models under the same settings, including a nearest-neighbour vote over the same graph, gradient-boosted trees and a random forest, over three seeds that each draw a new split, with ablations, leave-one-family-out and feature-noise tests. On CICIDS2017 the residual network reached a binary F1 of 98.11% (SD 0.18) and on NF-UNSW-NB15-v2 90.57% (0.38), indistinguishable from a multilayer perceptron without the graph and from other graph networks. Tree ensembles were 1.48 to 1.57 F1 points higher on CICIDS2017 and led on the area under the precision-recall curve on both datasets; their CICIDS2017 multiclass macro-F1 reached 93.08% against 64.51%, a gap mainly from benign flows the residual network assigned to small attack classes. No ablation, including transductive graph construction, changed F1 beyond seed variation. Under Gaussian feature noise (SD 0.5), the residual network kept an F1 of 96.16%, while XGBoost fell to 0.67%. Under a leakage-controlled protocol, message passing over flow graphs adds no measurable accuracy over a perceptron, and tree ensembles remain stronger on both; the advantage of graph models lies in robustness to feature noise.
BibTeX
@misc{iqbal2026leakage,
  author = {Zahid Iqbal},
  title = {{Leakage-free evaluation of residual graph convolutional networks for network intrusion detection}},
  year = {2026},
  eprint = {2611.04660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04660},
}
Figure