Inference-to-complete: A High-performance and Programmable Data-plane Co-processor for Neural-network-driven Traffic Analysis

Tong Yang, Dong Wen, Zhongpei Liu, Tong Yang 0003, Tao Li, Tianyun Li, Chenglong Li, Jie Li, Zhigang SunarXiv 2024 · DOI · arXiv

Introduction

As network speeds escalate toward 100 Gbps and beyond, the demand for intelligent data planes (IDPs) capable of real-time traffic analysis—such as malware detection and resource management—has intensified. However, existing IDP solutions face a critical "trilemma" involving flexibility, performance, and data-plane awareness. Most hardware accelerators are either too rigid to support evolving neural network (NN) architectures, introduce prohibitive latency into the packet forwarding path, or force the entire network switch to operate at a reduced clock speed, compromising overall throughput.

System Architecture

To overcome these challenges, we present Kaleidoscope, a high-performance, programmable co-processor that decouples inference from the main forwarding pipeline. Kaleidoscope utilizes a "bypass" architecture, mirroring traffic to a side-channel. This design allows the primary data plane to maintain its native high-speed clock while the co-processor operates on an optimized asynchronous clock, ensuring that standard network operations are never slowed down by complex inference tasks.

Figure 1: The Kaleidoscope system architecture, showing the bypass path and the heterogeneous inference engine.

As shown in the architecture overview, Kaleidoscope includes a traffic monitor that identifies flows and routes them to the appropriate inference path. Results are written back to a query table, which the data plane can access with a negligible overhead of approximately 15.5 ns.

Heterogeneous Inference Engine

Kaleidoscope recognizes that network traffic is non-uniform, typically following a power-law distribution where a small number of "elephant flows" account for the majority of volume, while numerous "mouse flows" are short-lived. To optimize resource allocation, Kaleidoscope employs a heterogeneous inference engine:

  • Fast Process Element (FPE): Optimized for "mouse flows," providing ultra-low latency inference using Matrix-Vector operations.
  • Heavy Process Element (HPE): Designed for "elephant flows," utilizing a systolic array-based engine to execute complex, high-accuracy models like CNNs or RNNs.
Figure 2: The architecture of the Heavy Process Element (HPE), featuring a systolic array for complex Matrix-Matrix computations.

Both elements utilize a Very-Long-Instruction-Word (VLIW) architecture, allowing them to support various NN types through firmware updates rather than hardware redesigns. Furthermore, by feeding raw packet bytes directly into the NNs, Kaleidoscope eliminates the heavy computational cost associated with traditional feature extraction.

Experimental Evaluation

We evaluated Kaleidoscope using the ISCX-VPN-2016 and USTC-TFC benchmarks for encrypted traffic analysis and malware detection. On an FPGA implementation, the system achieved a line-rate throughput of 100 Gbps, with simulations in 28nm ASIC technology reaching up to 1.6 Tbps.

Key performance metrics include:

  • Latency: Inference latencies for mouse flows ranged from 256 ns to 352 ns, significantly outperforming inline accelerators like Taurus.
  • Accuracy: By supporting larger models, Kaleidoscope achieved 92.8% to 99.2% accuracy, surpassing existing solutions by up to 26.5% in malware detection tasks.
  • Efficiency: The base FPGA configuration requires only 7% of logic resources (LUTs), preserving capacity for other critical network functions.

Conclusion

Kaleidoscope demonstrates that decoupling inference from forwarding via a bypass architecture is a viable path toward truly intelligent, high-speed data planes. By balancing the needs of mouse and elephant flows through heterogeneous processing elements, it achieves state-of-the-art accuracy without sacrificing network performance. Future work will explore the integration of hardware sketches to support statistical feature extraction and methods to mitigate hash collisions in high-concurrency scenarios.

Summary updated

BibTeX
@misc{yang2024inference,
  author = {Tong Yang and Dong Wen and Zhongpei Liu and Tong Yang and Tao Li and Tianyun Li and Chenglong Li and Jie Li and Zhigang Sun},
  title = {{Inference-to-complete: A High-performance and Programmable Data-plane Co-processor for Neural-network-driven Traffic Analysis}},
  year = {2024},
  eprint = {2411.00408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.00408},
}

Loading related content...

Figure