Abstract
As the size and source of network traffic increase, so does the challenge of monitoring and analysing network traffic. Therefore, sampling algorithms are often used to alleviate these scalability issues. However, the use of high entropy data streams, through the use of either encryption or compression, further compounds the challenge as current state of the art algorithms cannot accurately and efficiently differentiate between encrypted and compressed packets. In this work, we propose a novel traffic classification method named HEDGE (High Entropy DistinGuishEr) to distinguish between compressed and encrypted traffic. HEDGE is based on the evaluation of the randomness of the data streams and can be applied to individual packets without the need to have access to the entire stream. Findings from the evaluation show that our approach outperforms current state of the art. We also make available our statistically sound dataset, based on known benchmarks, to the wider research community.
Introduction
In the realm of network security, high entropy is typically a signature of obfuscated data. Whether a user is protecting privacy via AES encryption or a system is conserving bandwidth through GZIP compression, the resulting packets appear as stochastic noise to most monitoring tools. For security analysts and smart firewalls, however, the distinction between "encrypted" and "compressed" is critical: one represents a secure tunnel, while the other may be a potentially reversible data leak.
As network traffic volume explodes due to IoT and mobile devices, monitoring every bit becomes a scalability nightmare. Many systems resort to sampling, but current tools struggle to differentiate between high-entropy data streams at the packet level. This capability is essential for security auditing (identifying devices using compression instead of encryption), malware detection (spotting data exfiltration), and overall monitoring efficiency.
Methodology: The HEDGE Approach
HEDGE (High Entropy DistinGuishEr) is a novel, real-time traffic classification method that distinguishes between encrypted and compressed packets by analyzing their statistical randomness. Rather than relying on computationally intensive "black box" deep learning models, HEDGE utilizes a lightweight tree-based threshold system built on classical statistical randomness tests:
- Chi-Square Test: Evaluates if the bit distribution in a packet deviates from what is expected in a truly random sequence.
- NIST SP 800-22 Subset: Employs specific tests—Frequency within Block, Cumulative Sums, and Approximate Entropy—that are particularly effective at identifying subtle patterns left by compression algorithms.
- Gain Factor ((\gamma)): A tunable parameter allowing administrators to adjust detection strictness, balancing the trade-off between True Positives and False Negatives.
Unlike machine learning approaches with high computational overhead, HEDGE operates with linear complexity (O(n)), making it ideal for high-speed, real-time hardware implementation.
Experimental Setup and Datasets
The researchers developed a new, statistically sound dataset to avoid biases found in older benchmarks. Raw data was pulled from diverse sources including the COCO Dataset (images), Project Gutenberg (text), ArXiv (PDFs), and system files from Windows 10 and Ubuntu. These files were processed using various encryption (AES, Camelia) and compression (ZIP, RAR, BZIP2, GZIP) methods across packet sizes ranging from 1KB to 64KB.
Key Results and Insights
HEDGE consistently outperformed previous state-of-the-art methods. A key finding is that the "source" of the data matters: compressed text or binary files are much easier to distinguish from encrypted data than compressed images or videos, which already possess high intrinsic entropy.
As shown in the evaluation, accuracy scales with packet size. At 64KB, HEDGE achieved 94.72% accuracy. Even at the challenging 1KB packet size, HEDGE reached 70.6% accuracy, a significant improvement over the 66.9% achieved by previous state-of-the-art methods. Because it uses a threshold system rather than a neural network, HEDGE is easily parallelizable and requires no heavy training phase once initial thresholds are established.
Conclusion and Future Work
HEDGE represents a significant step forward in efficient traffic classification. While accuracy naturally decreases as packet sizes shrink—dropping to roughly 68.6% for 1KB packets when considering all file types—the method provides a scalable alternative to deep learning. Future work aims to refine the method for sub-1KB packets and explore identifying specific file types within compressed streams to further enhance proactive security measures.
Summary updated