Abstract
The Anonymized Network Sensing Graph Challenge provides a standardized framework and massive datasets (100B+ packets) to stimulate innovation in high-performance network security analysis. By representing network traffic as hypersparse matrices, the challenge enables researchers to develop privacy-preserving tools that can analyze global-scale internet traffic at speeds exceeding a billion packets per second.
Introduction
Protecting modern networks requires the analysis of vast datasets, often spanning multiple organizational boundaries. However, two significant hurdles impede progress: privacy and scale. Sharing raw network captures (PCAP files) poses severe privacy risks, while the sheer volume of data from "Internet telescopes"—which monitor darkspace or unused IP ranges—is too massive for traditional real-time processing tools.
The Anonymized Network Sensing Graph Challenge addresses these issues by formalizing the "Traffic Matrix" as a standard data product. By treating network flows as a graph—where IP addresses represent nodes and packets represent edges—researchers can leverage sparse linear algebra to perform complex security analytics. This challenge establishes a formal framework with curated datasets and performance metrics to foster the development of faster, more secure network sensing pipelines.
The Graph-Matrix Framework
The core innovation of this challenge is the application of graph-matrix duality to network traffic. Network traffic is inherently a graph, and hypersparse matrices provide the most efficient representation for such data. This approach allows the use of highly optimized linear algebra libraries, such as GraphBLAS, to solve complex networking problems.
A key property utilized is permutation invariance. Because matrix operations are generally invariant to the reordering of rows and columns, anonymizing IP addresses (effectively a permutation) does not alter the fundamental network statistics. This ensures that security threats, such as botnets, can be identified in anonymized data as effectively as in raw data.
Methodology and Pipeline
The challenge defines a rigorous six-step pipeline for processing network data:
- Ingestion: Reading massive PCAP files, typically containing $2^{30}$ packets.
- Extraction: Pulling source and destination IP addresses from packet headers.
- Anonymization: Relabeling IPs consistently (e.g., using CryptoPAN) to preserve privacy while maintaining structural properties.
- Construction: Building sequential traffic matrices ($A_t$), where each entry $(i, j)$ represents the packet count between source $i$ and destination $j$.
- Storage: Saving matrices in efficient formats like Compressed Sparse Row (CSR) or SuiteSparse GraphBLAS.
- Analysis: Aggregating matrices via binary summation and computing network quantities such as "fan-out," "unique destinations," and "link packets."
To simplify statistical analysis and ensure consistent computational load, the challenge utilizes constant packet windows ($N_V = 2^{17}$) per matrix rather than time-based divisions. Furthermore, hierarchical aggregation—summing matrices in a binary tree fashion—allows for efficient traffic analysis across multiple time scales without re-processing raw packets.
Datasets
The challenge incorporates three distinct data types to provide a comprehensive testing ground:
- Randomized Data: $2^{30}$ packets generated via a pseudo-random number generator (PCG32) to establish a performance baseline.
- CAIDA Telescope Data: Real-world, anonymized traffic from the CAIDA "darknet" telescope, representing unsolicited traffic across a massive /8 network.
- GreyNoise Enrichment: A metadata layer identifying known malicious scanners and "honeyfarm" activity, allowing for the validation of analysis against known threats.
Results and Performance
The reference implementation, utilizing C and GraphBLAS, demonstrates significant performance capabilities:
- Linear Scaling: The pipeline shows near-perfect linear scaling across CPU cores.
- High Throughput: On dual Intel Xeon Platinum nodes, processing rates reached between $10^8$ and $10^9$ packets per second.
- Memory Efficiency: A streaming approach enables the construction of traffic matrices with a small memory footprint, allowing terabytes of compressed PCAP data to be processed on standard server hardware.
One noted bottleneck is in-line anonymization, which requires optimization (such as look-up tables) to match the speed of matrix construction.
Limitations and Future Work
While the current challenge focuses on the 32-bit IPv4 space, extending these hypersparse techniques to the 128-bit IPv6 space is a primary goal. Although the addressable matrix size increases significantly, the number of active entries remains manageable. Future work will also explore "semantically secure" anonymization methods to provide stronger privacy guarantees and the integration of matrix construction directly into network switches or specialized hardware for real-edge processing.
Summary updated