TL;DR
This paper introduces CARAVAN, a system that enables in-network machine learning (ML) models to adapt to changing network conditions through online learning. It tackles the challenge of labeling network traffic without ground truth by repurposing existing systems and foundation models, and efficiently triggers retraining using an "accuracy proxy" metric.
What Problem Does This Paper Address?
Modern networks are dynamic and complex, leading to fluctuating traffic patterns and evolving behaviors. While in-network ML models offer performance benefits, they are typically trained offline and struggle to adapt to these changes, resulting in performance degradation. Existing online learning systems often rely on readily available labels or manual labeling, which are impractical for high-throughput network environments. This paper addresses the critical need for a practical system that can automatically label evolving network traffic and efficiently retrain in-network ML models to maintain performance in real-time.
Methods
CARAVAN introduces a novel online learning framework for in-network ML models, comprising three key components:
- Labeling Agent: This component is responsible for generating labels for incoming network traffic. It achieves this by leveraging various "knowledge sources," which can be existing network systems (like heuristics or access control lists) or powerful foundation models (like GPT-4 or Gemini). These sources provide labels, and a majority voting mechanism is used to determine the final generated label.
- Model Validation: This stage monitors the performance of the in-network ML model. It introduces a new metric called "accuracy proxy," which uses the generated labels as a reference to estimate the model's accuracy. This proxy is crucial for detecting performance degradation without requiring actual ground truth labels.
- Retraining Trigger: Based on the accuracy proxy, this component decides when to initiate retraining. It uses an "accuracy-based" trigger that signals retraining when the accuracy proxy drops significantly, indicating potential concept or data drift. This prevents unnecessary retraining and saves computational resources.
CARAVAN also incorporates a "rule cache" mechanism for expensive knowledge sources like LLMs, allowing them to generate rules for faster, cheaper labeling for a limited duration.
Datasets
The paper evaluates CARAVAN using the following datasets:
- CIC-IDS2017: Used for network intrusion detection.
- UNSW-NB15: Also used for network intrusion detection.
- UNSW-IoT: Used for IoT traffic classification.
These datasets are standard in prior work on in-network ML and are used to simulate network drifts by introducing new attack traffic or unseen IoT device traffic.
Key Insights and Ideas
- Repurposing Existing Systems for Labeling: The paper demonstrates that seemingly unsuitable systems like heuristics and access control lists can be effectively repurposed as "knowledge sources" to generate valuable labels for in-network ML models.
- Foundation Models as Labeling Agents: Large language models (LLMs) can be adapted to act as powerful labeling agents, providing high-quality labels, though at a higher computational cost.
- Accuracy Proxy for Drift Detection: The "accuracy proxy" metric provides a practical and effective way to monitor model performance degradation and detect concept/data drifts without relying on ground truth labels.
- Selective Retraining Saves Resources: By using accuracy proxy and retraining triggers, CARAVAN can significantly reduce the computational cost of online learning by retraining only when necessary, rather than continuously.
- Rule Caching for Cost Efficiency: For expensive LLM-based labeling, generating and caching rules can drastically reduce API costs and improve efficiency without significant accuracy loss.
Key Results
- Improved Accuracy: CARAVAN achieved a 30.3% improvement in F1 score on average compared to static offline models across three applications.
- Reduced Computational Cost: CARAVAN reduced GPU compute time for retraining by an average of 61.3% compared to continuous learning systems. In simulations, this saving reached up to 74.55%.
- Comparable Accuracy to Continuous Learning: CARAVAN sustained comparable inference accuracy to continuous learning systems while consuming significantly less resources.
- FPGA Testbed Performance: On an end-to-end Taurus FPGA testbed, CARAVAN maintained high inference accuracy at network line-rate, improving F1 score by an average of 30% with 56.23% less CPU usage and similar memory footprint compared to continuous retraining baselines.
- Labeling Effectiveness: Noisy and weak supervision labels generated by imperfect knowledge sources still led to comparable accuracy gains as ground truth labels.
Limitations and Future Work
- Optimizing Sample Selection: CARAVAN currently uses random sampling for input data. Future work could focus on developing more efficient and effective algorithms for online sample selection, especially for heavy-tailed network traffic.
- Reverting Models: The system currently focuses on updating models. Future research could explore mechanisms to revert or reset models to a previous good state, especially in scenarios where training data might be compromised by malicious actors (data poisoning).
- Data Collection and Generation: More research is needed on optimal strategies for collecting and sampling network data specifically for retraining ML models, considering dynamic sampling rates based on detected events.
- Domain-Specific Knowledge Sources: While GPT-4 was used as a proof-of-concept, developing specialized foundation models for networking and cybersecurity could further improve labeling accuracy and efficiency.
- Generalizing to Larger Models: The insights from CARAVAN could be generalized to larger ML models deployed in the control plane, which also face similar challenges of concept drift and lack of labels.
Summary updated