Anonymized Network Sensing Graph Challenge

Hayden Jananthan, Michael Jones, William Arcand, David Bestor, William Bergeron, Daniel Burrill, Aydin Buluç, Chansup Byun, Timothy Davis, Vijay Gadepally, Daniel Grant, Matthew Hubbell, Piotr Luszczek, Peter Michaleas, Lauren Milechin, Chasen Milner, Guillermo Morales, Andrew Morris, Julie Mullen, Ritesh Patel, Alex Pentland, Sandeep Pisharody, Andrew Prout, Albert Reuther, Antonio Rosa, Gabriel Wachman, Charles Yee, Jeremy Kepner

Abstract

The Anonymized Network Sensing Graph Challenge provides a standardized framework and massive datasets (100B+ packets) to stimulate innovation in high-performance network security analysis. By representing network traffic as hypersparse matrices, the challenge enables researchers to develop privacy-preserving tools that can analyze global-scale internet traffic at speeds exceeding a billion packets per second.

Introduction

Protecting modern networks requires the analysis of vast datasets, often spanning multiple organizational boundaries. However, two significant hurdles impede progress: privacy and scale. Sharing raw network captures (PCAP files) poses severe privacy risks, while the sheer volume of data from "Internet telescopes"—which monitor darkspace or unused IP ranges—is too massive for traditional real-time processing tools.

The Anonymized Network Sensing Graph Challenge addresses these issues by formalizing the "Traffic Matrix" as a standard data product. By treating network flows as a graph—where IP addresses represent nodes and packets represent edges—researchers can leverage sparse linear algebra to perform complex security analytics. This challenge establishes a formal framework with curated datasets and performance metrics to foster the development of faster, more secure network sensing pipelines.

The Graph-Matrix Framework

The core innovation of this challenge is the application of graph-matrix duality to network traffic. Network traffic is inherently a graph, and hypersparse matrices provide the most efficient representation for such data. This approach allows the use of highly optimized linear algebra libraries, such as GraphBLAS, to solve complex networking problems.

A key property utilized is permutation invariance. Because matrix operations are generally invariant to the reordering of rows and columns, anonymizing IP addresses (effectively a permutation) does not alter the fundamental network statistics. This ensures that security threats, such as botnets, can be identified in anonymized data as effectively as in raw data.

Methodology and Pipeline

The challenge defines a rigorous six-step pipeline for processing network data:

  1. Ingestion: Reading massive PCAP files, typically containing $2^{30}$ packets.
  2. Extraction: Pulling source and destination IP addresses from packet headers.
  3. Anonymization: Relabeling IPs consistently (e.g., using CryptoPAN) to preserve privacy while maintaining structural properties.
  4. Construction: Building sequential traffic matrices ($A_t$), where each entry $(i, j)$ represents the packet count between source $i$ and destination $j$.
  5. Storage: Saving matrices in efficient formats like Compressed Sparse Row (CSR) or SuiteSparse GraphBLAS.
  6. Analysis: Aggregating matrices via binary summation and computing network quantities such as "fan-out," "unique destinations," and "link packets."

To simplify statistical analysis and ensure consistent computational load, the challenge utilizes constant packet windows ($N_V = 2^{17}$) per matrix rather than time-based divisions. Furthermore, hierarchical aggregation—summing matrices in a binary tree fashion—allows for efficient traffic analysis across multiple time scales without re-processing raw packets.

Datasets

The challenge incorporates three distinct data types to provide a comprehensive testing ground:

  • Randomized Data: $2^{30}$ packets generated via a pseudo-random number generator (PCG32) to establish a performance baseline.
  • CAIDA Telescope Data: Real-world, anonymized traffic from the CAIDA "darknet" telescope, representing unsolicited traffic across a massive /8 network.
  • GreyNoise Enrichment: A metadata layer identifying known malicious scanners and "honeyfarm" activity, allowing for the validation of analysis against known threats.

Results and Performance

The reference implementation, utilizing C and GraphBLAS, demonstrates significant performance capabilities:

  • Linear Scaling: The pipeline shows near-perfect linear scaling across CPU cores.
  • High Throughput: On dual Intel Xeon Platinum nodes, processing rates reached between $10^8$ and $10^9$ packets per second.
  • Memory Efficiency: A streaming approach enables the construction of traffic matrices with a small memory footprint, allowing terabytes of compressed PCAP data to be processed on standard server hardware.

One noted bottleneck is in-line anonymization, which requires optimization (such as look-up tables) to match the speed of matrix construction.

Limitations and Future Work

While the current challenge focuses on the 32-bit IPv4 space, extending these hypersparse techniques to the 128-bit IPv6 space is a primary goal. Although the addressable matrix size increases significantly, the number of active entries remains manageable. Future work will also explore "semantically secure" anonymization methods to provide stronger privacy guarantees and the integration of matrix construction directly into network switches or specialized hardware for real-edge processing.

Summary updated

BibTeX
@inproceedings{jananthan2024anonymized,
  author = {Hayden Jananthan and Michael Jones and William Arcand and David Bestor and William Bergeron and Daniel Burrill and Aydin Buluç and Chansup Byun and Timothy Davis and Vijay Gadepally and Daniel Grant and Michael Houle and Matthew Hubbell and Piotr Luszczek and Peter Michaleas and Lauren Milechin and Chasen Milner and Guillermo Morales and Andrew Morris and Julie Mullen and Ritesh Patel and Alex Pentland and Sandeep Pisharody and Andrew Prout and Albert Reuther and Antonio Rosa and Gabriel Wachman and Charles Yee and Jeremy Kepner},
  title = {{Anonymized Network Sensing Graph Challenge}},
  booktitle = {HPEC},
  year = {2024},
  doi = {10.1109/hpec62836.2024.10938508},
}
Figure