Direct Batch Growth Hierarchical Self-Organizing Mapping Based on Statistics for Efficient Network Intrusion Detection

Xiaofei Qu, Lin Yang, Kai Guo, Meng Sun, Linru Ma, Tao Feng, Shuangyin Ren, Kechao Li, Xin MaIEEE Access 2020 · DOI

Introduction

Modern network security faces a daunting "needle in a haystack" problem. As network traffic becomes increasingly massive, high-dimensional, and asymmetrically distributed, traditional signature-based systems are struggling to keep pace with "zero-day" threats and evolving attack vectors. While unsupervised learning—specifically Self-Organizing Maps (SOM)—offers a powerful way to visualize and cluster unknown data, standard models often fail to capture the inherently hierarchical nature of network traffic.

Existing models like the Directed Batch Growth SOM (DBGSOM) can become computationally unwieldy as data volume grows, leading to high error rates. A significant challenge is "bunched" data: different types of traffic often end up clustered in the same neuron because their Euclidean distances are too similar, making them indistinguishable to a standard SOM. This paper introduces the Statistic-Enhanced Direct Batch Growth Hierarchical SOM (SE-DBGHSOM) to address these limitations through vertical expansion and frequency-domain analysis.

The SE-DBGHSOM Architecture

The core innovation of SE-DBGHSOM lies in its ability to grow both horizontally and vertically, adapting its granularity to the complexity of the data. Unlike standard 2D maps, this model utilizes a hierarchical growth mechanism along a Z-axis.

Hierarchical Growth and Statistical Thresholding

When a specific neuron's cumulative error exceeds a dynamically calculated vertical threshold ($T$), the model generates a new sub-map specifically for the data associated with that neuron. This allows for multi-level granularity: the top layers provide coarse classification, while deeper layers perform fine-grained analysis on complex data clusters.

The hierarchical growth process of the SE-DBGHSOM, showing how neurons expand into sub-maps across multiple layers based on error thresholds.

Crucially, these growth thresholds are not static. They are dynamically calculated based on the standard deviation and statistical properties of the input data. This makes the model "case-sensitive," allowing it to adapt its structure to the specific characteristics of the dataset it is analyzing.

Resolving Data Ambiguity with Fourier Analysis

To solve the problem of "bunched data" where Euclidean distance fails, the authors integrated a Fourier Model. When the SOM cannot separate clusters in the spatial domain, the model applies Fourier transforms to calculate Cross-Correlation (CC) and Auto-Correlation (AC).

By introducing the Correlation Distribution Factor (CDF), the researchers defined the range of correlation values for specific attack types. This frequency-domain analysis reveals patterns in packet behavior—such as the timing and sequence of requests—that are invisible in the raw feature space. This allows the system to distinguish between normal traffic and sophisticated attacks that mimic legitimate behavior.

Performance Evaluation

The model was validated using three prominent benchmarks: KDD99, the Moore Dataset (Cambridge University traffic), and CICIDS2017.

Key Results

  • High Precision: On the KDD99 dataset, the SE-DBGHSOM achieved a near-unity 99.85% Maximal Detection Rate (DR) with a remarkably low 0.19% False Positive Rate (FR).
  • Efficiency: The hierarchical structure prevents the map from becoming excessively large and redundant, significantly reducing training time compared to standard DBGSOM.
  • Fine-Grained Success: The Fourier-based CDF successfully distinguished specific attacks like snmpgetattack (an R2L attack) with high probability, even when those attacks were initially entangled with normal traffic.
The Correlation Distribution Factor (CDF) for the snmpgetattack, demonstrating how frequency-domain analysis distinguishes specific attack patterns.

In the Moore dataset, the model outperformed traditional supervised methods like Naive Bayes across almost all traffic categories, particularly in identifying MAIL and DATABASE traffic, proving the superiority of an unsupervised approach enhanced with statistical fine-tuning.

Conclusion and Future Work

SE-DBGHSOM represents a significant step forward in making SOMs practical for modern network environments. By spending computational resources only on expanding "complex" clusters while leaving "simple" clusters at higher levels, the model achieves both efficiency and accuracy.

However, challenges remain. Some minority attacks (e.g., saint, satan) still show lower detection rates due to overlapping correlation ranges. Future work will explore integrating Information Theory, such as entropy-based metrics, to better handle distributed asymmetric data. Additionally, further research is needed to optimize the computational overhead of performing Fourier transforms on-the-fly in high-speed, real-time production environments.

Summary updated

BibTeX
@article{qu2020direct,
  author = {Xiaofei Qu and Lin Yang and Kai Guo and Meng Sun and Linru Ma and Tao Feng and Shuangyin Ren and Kechao Li and Xin Ma},
  title = {{Direct Batch Growth Hierarchical Self-Organizing Mapping Based on Statistics for Efficient Network Intrusion Detection}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2976810},
}
Figure