TL;DR
This paper evaluates how effectively Large Language Models (LLMs) can reason over network traffic traces. By comparing raw textual dumps to protocol-filtered data, the researchers found that "less is more": protocol-aware filtering drastically reduces computational overhead while significantly boosting the accuracy of model responses.
What Problem Does This Paper Address?
Network traffic analysis is the backbone of cybersecurity and troubleshooting, but it’s a grueling task. Analysts must sift through massive packet captures (PCAPs), reconstruct complex communication flows, and identify subtle patterns—all of which require deep technical expertise.
While LLMs are being touted as potential "AI assistants" for this work, we don't actually know how reliable they are. Can an LLM accurately count packets? Can it tell if a TCP handshake actually finished, or is it just hallucinating based on technical-sounding keywords? This paper establishes a reproducible methodology to test these capabilities, focusing on whether the representation of the data (raw vs. filtered) changes the model's ability to "understand" the network.
Methods
The authors developed a structured pipeline to move from raw network data to evaluated AI responses:
- Data Preprocessing: They used
TSharkto convert PCAP files into two textual "views": - Default View: A standard one-line-per-packet summary.
- Protocol-Filtered View: A concise representation containing only the specific fields relevant to the protocol being questioned (e.g., just TCP flags and sequence numbers for a TCP-related query).
- Question-Answer Generation: They created a ground-truth dataset of 19 questions ranging from simple field extraction (MAC addresses) to complex flow interpretation (identifying failed handshakes).
- LLM-as-a-Judge: To avoid the slow, subjective nature of human grading, they used a high-parameter model (Llama 3.3 70B) to grade the answers from the smaller models. This "judge" was validated against human experts to ensure its reliability.
The study evaluated four models: Llama 3.1 (8B and 70B) and Qwen 2.5 (7.6B and 72B), using a zero-shot prompting strategy.
Datasets
The study utilized the CICIDS2017 dataset, a well-known benchmark in the networking community. Specifically, they used the "Monday" capture, which contains 50,000 packets of benign (normal) traffic, to test the models' foundational understanding before introducing malicious anomalies.
Key Insights and Ideas
- The "Noise" Tax: As the number of input packets increases, LLM accuracy generally decreases. This suggests that even within a model's context window, irrelevant packets act as "noise" that distracts the model from the relevant data points.
- Filtering is a Superpower: Protocol-aware filtering achieved a median token reduction of up to 79.8%. More importantly, it improved the "transition score" (a measure of qualitative improvement) in 11 out of 12 experimental scenarios.
- State Tracking is the Final Boss: LLMs are surprisingly good at looking up a single MAC address, but they struggle significantly with "stateful" reasoning—such as determining if a connection was refused or ignored during a port scan.
- Hallucinating "None": One of the most interesting failure patterns was the models' inability to handle empty results. When a protocol was absent from a trace, models often hallucinated service ports or invented network flows rather than simply stating "None."
Key Results
- Top Performer: Qwen 2.5 72B emerged as the most capable model, achieving the highest correctness scores across both default and filtered views.
- Judge Accuracy: The LLM-as-a-judge approach showed a 95.1% agreement with human annotators, proving that automated pipelines can reliably evaluate technical reasoning.
- Quantitative Boost: For Llama 3.1 70B at a 100% context budget, protocol filtering improved the qualitative "transition score" by +15.3%, effectively turning many "Incorrect" or "Partial" answers into "Correct" ones.
Limitations and Future Work
The authors acknowledge three main hurdles:
- Static Views: The current filtering is pre-defined. Future work should involve agentic environments where the LLM can dynamically choose which filters to apply based on the user's question.
- Context Constraints: Even with filtering, very large traces exceed current context windows.
- Cross-Layer Dependencies: The current filtered views sometimes strip out information (like IP addresses in a TCP-only view) that is necessary to distinguish between concurrent connections using the same ports.
By moving toward "intent-based" preprocessing, the researchers hope to make LLMs truly reliable partners for network security analysts.
Summary updated