Likelihood-Based Inference for Modelling Packet Transit From Thinned Flow Summaries

Prosha Rahman, Boris Beranger, Scott A. Sisson, Matthew Roughan

Abstract

Network traffic speeds and volumes present practical challenges to analysis. Packet thinning and flow aggregation protocols provide smaller structured data summaries, but conversely impede statistical inference. Methods which model traffic propagation typically do not account for the packet thinning and aggregation in their analysis and are of limited practical use. We introduce a likelihood-based analysis which fully incorporates packet thinning and flow aggregation. Inferences can hence be made for models on the level of individual packets while only observing thinned flow summaries. We establish consistency of the resulting maximum likelihood estimator, derive bounds on the volume of traffic which should be observed to achieve a desired degree of efficiency, and identify an ideal family of models. The robust performance of the estimator is examined through simulated analyses and an application on a publicly accessible trace which captured in excess of 36m packets over a 1 minute period.

Introduction

In modern high-speed networking, the sheer volume of data makes recording every individual packet practically impossible. To manage this "firehose" of information, network engineers rely on two primary data reduction strategies: packet thinning (randomly sampling a fraction of packets, such as 1 out of 1,000) and flow aggregation (grouping packets into "flows" and recording summary statistics like start time, duration, and total packet count).

While these strategies are essential for storage and transmission, they create a significant "statistical fog." Traditional inference methods often ignore the thinning process or rely on simplistic "method-of-moments" estimators. These traditional approaches frequently fail to converge or provide accurate results when traffic patterns become complex, particularly for the "mouse flows" (short-lived connections) that dominate internet traffic. This paper introduces a rigorous, likelihood-based framework designed to perform packet-level inference using only these thinned and summarized data streams.

Methodology

Symbolic Data Analysis (SDA)

The core of the proposed framework is Symbolic Data Analysis (SDA). Unlike traditional statistics that treat individual data points as the unit of analysis, SDA treats data summaries—such as intervals, histograms, or flow records—as the fundamental units. This allows the researchers to model the generative process of the summary itself.

The NetFlow Likelihood

The authors derive a mathematical representation of a NetFlow summary that accounts for the underlying packet behavior. In this model, the observed flow duration is treated as the sum of unobserved inter-packet arrival times. To account for Bernoulli sampling (thinning), where each packet has a probability $q$ of being recorded, the framework integrates over all possible "latent" flows that could have produced the observed thinned summary.

Traffic Modelling

The traffic is represented using a Bartlett–Lewis Model, a cluster renewal process where "main" processes (the arrival of new flows) trigger "subsidiary" processes (the arrival of packets within those flows). By applying the likelihood-based approach to this model, the researchers provide the first formal proof that an SDA-based maximum likelihood estimator (MLE) attains the same consistency and asymptotic normality as estimators derived from full, unsummarized datasets.

Experimental Evaluation

Synthetic Simulations

The method was first tested on synthetic networks using Zeta distributions for flow sizes—to mimic the heavy-tailed nature of internet traffic—and Gamma distributions for packet arrivals. The simulations demonstrated that the NetFlow MLE converged to true parameters even under extreme thinning ($q = 0.0001$). In contrast, traditional moments-based methods often failed to provide valid estimates under these conditions.

Real-World Application: CAIDA Trace

The framework was applied to a publicly available CAIDA (Center for Applied Internet Data Analysis) dataset from 2019. This trace contained over 36 million packets captured over a one-minute period from a high-speed backbone link. The NetFlow MLE provided a satisfactory fit for packet transit patterns using a dataset 26 times smaller than the full packet trace, demonstrating significant efficiency gains in data storage without sacrificing inferential power.

Key Insights

  • The "Ideal" Family: The research identifies that if the packet arrival model belongs to the Natural Exponential Family (NEF), the NetFlow aggregation process loses zero inferential power compared to having the full data. This provides a clear mathematical criterion for selecting traffic models.
  • Information vs. Storage Trade-off: The authors derived a "lower bound" formula to calculate the number of summarized flows required to match the accuracy of a full-packet dataset. This allows engineers to make informed decisions about storage requirements versus statistical confidence.
  • Computational Costs: While highly accurate, the likelihood method is computationally intensive. Calculating the convolutions of probability densities for "elephant flows" (very large flows) is expensive. For example, computing the MLE on 4,553 thinned flows took approximately 3.8 hours, whereas simpler arithmetic estimators are nearly instantaneous.

Conclusion and Future Work

This likelihood-based framework bridges the gap between high-level flow summaries and low-level packet behavior. While it offers superior robustness and statistical consistency over traditional methods, the computational overhead remains a challenge for very large flows. Future research will focus on refining approximations for distributions without closed-form solutions (like Log-Normal) and extending the model to account for dependent inter-packet arrival processes.

Summary updated

Abstract

The substantial growth of network traffic speed and volume presents practical\nchallenges to network data analysis. Packet thinning and flow aggregation\nprotocols such as NetFlow reduce the size of datasets by providing structured\ndata summaries, but conversely this impedes statistical inference. Methods\nwhich aim to model patterns of traffic propagation typically do not account for\nthe packet thinning and summarisation process into the analysis, and are often\nsimplistic, e.g.~method-of-moments. As a result, they can be of limited\npractical use.\n We introduce a likelihood-based analysis which fully incorporates packet\nthinning and NetFlow summarisation into the analysis. As a result, inferences\ncan be made for models on the level of individual packets while only observing\nthinned flow summary information. We establish consistency of the resulting\nmaximum likelihood estimator, derive bounds on the volume of traffic which\nshould be observed to achieve required levels of estimator accuracy, and\nidentify an ideal family of models. The robust performance of the estimator is\nexamined through simulated analyses and an application on a publicly available\ntrace dataset containing over 36m packets over a 1 minute period.\n
BibTeX
@article{rahman2022likelihood,
  author = {Prosha Rahman and Boris Beranger and Scott A. Sisson and Matthew Roughan},
  title = {{Likelihood-Based Inference for Modelling Packet Transit From Thinned Flow Summaries}},
  journal = {IEEE Trans. Signal Inf. Process. Over Networks},
  year = {2022},
  doi = {10.1109/tsipn.2022.3188457},
}
Figure