Network Traffic Sampling System Based on Storage Compression for Application Classification Detection

Shichang Xuan, Dezhi Tang, Ilyong Chung, Youngju Cho, Xiaojiang Du, Wu YangIEEE Access 2020 · DOI

Abstract

Accurate application classification in high-speed networks is essential for modern network management but often faces significant memory and computational overhead. This paper presents MiniSamp, a high-performance traffic sampling system designed to reduce hardware resource consumption without sacrificing classification accuracy. By integrating shared counter trees for storage compression and adaptive search trees for accelerated flow lookups, MiniSamp addresses the bottlenecks of existing sampling methods in high-concurrency environments.

Introduction

To effectively manage modern networks, operators must classify traffic to distinguish between various application types, such as P2P file sharing and web browsing. However, high-speed networks exhibit a "heavy-tailed" distribution: a massive number of concurrent flows where a few "elephant" flows carry the majority of data, while millions of "mice" flows carry very little.

Existing sampling methods, such as RelSamp, attempt to retain sufficient features for machine learning classifiers but suffer from two primary bottlenecks:

  1. Storage Waste: Fixed-size counters are allocated for every flow, leading to significant memory waste on the millions of small flows.
  2. Search Inefficiency: Hash collisions often result in deep linear "collision chains" (linked lists), which slow down real-time flow tracking.

System Architecture and Methods

MiniSamp optimizes the traffic sampling pipeline by introducing a more efficient storage and search structure. The system captures mirrored traffic at the gateway, tracks flows in a specialized table, and stores features in a compressed format for subsequent classification.

The MiniSamp system architecture, illustrating the path from mirrored traffic sampling to final classification results.

Core Innovations

1. Shared Counter Trees

Instead of dedicated large counters for every flow, MiniSamp utilizes a hierarchical tree structure where multiple flows share enhanced counters. This structure exploits overlaps in flow data to represent statistical features (e.g., packet counts and flag frequencies) using significantly less memory than traditional flat tables.

2. Adaptive Non-Strictly Splay Trees

To resolve hash collision delays, MiniSamp replaces linear collision chains with search trees. While standard splay trees reorganize on every access—consuming excessive CPU—MiniSamp employs a "non-strictly" splay approach. It only reorganizes (splays) the tree when it detects a "hot" flow (one with continuous packet arrivals). This heuristic keeps active flows near the root for rapid access while minimizing computational overhead for transient flows.

Comparison of the hash flow tracking table using a search tree versus a traditional conflict chain.

Experimental Evaluation

The researchers evaluated MiniSamp using real-world traffic from a laboratory LAN gateway, utilizing 20 GB for training (labeled via Deep Packet Inspection) and 9 GB for testing.

Key Results

  • Storage Efficiency: MiniSamp saved up to 268 kB of flow table storage compared to standard methods and consistently required at least 111 kB less space than RelSamp.
  • Search Performance: The average search time for a flow node in MiniSamp was reduced to 73.2% of the time required by a standard hash list and 86.4% of a standard splay tree.
  • Classification Accuracy: Despite aggressive compression, the accuracy drop was negligible. Using a C4.5 classifier, accuracy only decreased by 0.2% to 1.9% compared to uncompressed baselines, proving that machine learning models are robust to the minor noise introduced by shared counters.

Conclusion

MiniSamp demonstrates that strategic storage compression and adaptive search structures can significantly alleviate the hardware constraints of high-speed network monitoring. By focusing on "hot flows" and hierarchical counters, the system achieves high-speed processing suitable for modern NICs and routers. Future work will focus on making system parameters, such as the splaying threshold and detection windows, self-adaptive to real-time traffic fluctuations.

Summary updated

BibTeX
@article{xuan2020network,
  author = {Shichang Xuan and Dezhi Tang and Ilyong Chung and Youngju Cho and Xiaojiang Du and Wu Yang},
  title = {{Network Traffic Sampling System Based on Storage Compression for Application Classification Detection}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2984258},
}
Figure