A new dynamic security defense system based on TCP_REPAIR and deep learning

Jianxun Tang, Mingsong Chen, Haoyu Chen, Shenqi Zhao, Yu HuangJ. Cloud Comput 2023 · OpenReview

Abstract

Honeypot is an active defense mechanism, which attracts attackers to interact with virtual resources in the honeypot mainly by simulating real working scenarios and deploying decoy targets, so as to prevent real resources from being damaged and collect attackers’ attack processes and analyze potential system vulnerabilities to proactively respond to similar attacks. Because of the existing honeypot system has defects such as the inability to deploy specific honeypots to induce attacks based on complex attacks, the inability to select the best honeypot for dynamic response based on honeypot deployment and maintenance costs during attack interactions, and insufficient ability to identify variants of known attack methods. Although hybrid honeypots can solve some of these problems by deploying low-interaction honeypots and high-interaction honeypots, they cannot really be applied to real production scenarios because of their slow TCP connection switching speed and inability to efficiently identify encrypted malicious traffic. In this paper, we propose a new dynamic security defense system based on the combination of TCP_REPAIR-based dynamic honeypot selection architecture and a deep learning-based intelligent firewall. The system accurately distributes encrypted or non-encrypted attack traffic and its variants through the intelligent firewall. The normal traffic is sent to the actual system, and the marked malicious traffic dynamically selects honeypots to respond according to the attack process. The experimental result indicated that the system can select honeypots for targeted responses according to the actual network situation quickly and dynamically and covertly, effectively improving the utilization rate of honeypot clusters as well as the ability to decoy.

Introduction

Traditional network defense mechanisms, such as firewalls and Intrusion Detection Systems (IDS), are increasingly challenged by two major trends: the ubiquity of traffic encryption and the sophistication of attackers who can detect sandbox environments. Most modern malicious traffic is encrypted (e.g., via HTTPS or VPNs), rendering traditional deep packet inspection (DPI) ineffective. Furthermore, hybrid honeypots—which transition attackers from low-interaction decoys to high-interaction systems for deeper analysis—often suffer from high redirection latency. Using Network Address Translation (NAT) or packet replay creates timing lags that tip off sophisticated attackers.

This paper presents a significant step forward in active network defense by combining deep learning with low-level Linux kernel features to create a "stealthy" honeypot system. By integrating a 1-Dimensional Convolutional Neural Network (1D-CNN) with the TCP_REPAIR utility, the authors have built a defense mechanism that can identify encrypted threats and redirect attackers to decoys without them ever realizing they’ve left the real network.

System Architecture

The proposed system architecture leverages Software-Defined Networking (SDN) to provide granular control over traffic flows. As shown in Figure 1, the system integrates several key components: a client, an IDS (Snort), a 1D-CNN-based intelligent firewall, a Ryu SDN controller, and a TCP_REPAIR proxy.

The proposed dynamic security defense system architecture, showing the integration of the 1D-CNN, Ryu controller, and TCP_REPAIR proxy.

The system operates by intercepting traffic at the network edge. Normal traffic is routed to the real system, while suspicious or malicious traffic is redirected to a honeypot cluster (HFish), which contains both High-Interaction Honeypots (HIH) and Low-Interaction Honeypots (LIH).

Methodology

The Intelligent Firewall (1D-CNN)

To address the challenge of encrypted malicious traffic, the authors developed an intelligent firewall based on a 1D-CNN. Unlike 2D-CNNs, which treat traffic like images and are computationally expensive, 1D-CNNs are designed to extract temporal features from data sequences efficiently. This module classifies traffic into three categories: normal, encrypted-malicious, or non-encrypted-malicious.

Flowchart of the CNN model training and prediction process for traffic classification.

By treating traffic as a 1D sequence, the system achieves high accuracy without the massive overhead required for image-based processing, making it suitable for high-speed network links.

Covert Redirection via TCP_REPAIR

The most novel technical contribution of this work is the use of the Linux kernel's TCP_REPAIR mode for covert redirection. When the system decides to migrate an attacker's connection to a high-interaction honeypot, it uses TCP_REPAIR to "hibernate" a TCP socket. This allows the system to extract the socket's internal state—including sequence numbers and window sizes—and recreate it on the decoy system. This approach eliminates the need for a new three-way handshake, removing the primary signature attackers use to detect honeypot transitions.

Experimental Evaluation

The system was evaluated using the ISCX dataset, a standard benchmark containing 12 categories of traffic, and custom SMTP scripts designed to test real-world active defense.

Key Results

  • Detection Accuracy: The 1D-CNN achieved an accuracy of 92.3% in identifying malicious encrypted traffic, significantly outperforming traditional machine learning models like C4.5.
  • Ultra-Low Latency: The redirection latency using TCP_REPAIR was approximately 0.013ms. This is nearly 100x faster than existing state-of-the-art systems like HoneyDOC (1.25ms) and Honeyd (0.86ms), making the transition virtually undetectable.
  • Throughput: The full system maintained a throughput of 583.7 Mbit/s. While the 1D-CNN introduces a ~21% overhead compared to clean communication, it remains highly performant for SDN-based architectures.

Conclusion

The integration of deep learning and kernel-level socket manipulation provides a robust and stealthy defense against modern encrypted threats. By achieving near-zero redirection latency and high detection accuracy, the proposed system effectively improves the decoy capabilities of honeypot clusters. Future work will focus on optimizing the 1D-CNN to reduce its computational bottleneck and testing the system against zero-day exploits in high-noise environments.

Summary updated

Abstract

Honeypot is an active defense mechanism, which attracts attackers to interact with virtual resources in the honeypot mainly by simulating real working scenarios and deploying decoy targets, so as to prevent real resources from being damaged and collect attackers’ attack processes and analyze potential system vulnerabilities to proactively respond to similar attacks. Because of the existing honeypot system has defects such as the inability to deploy specific honeypots to induce attacks based on complex attacks, the inability to select the best honeypot for dynamic response based on honeypot deployment and maintenance costs during attack interactions, and insufficient ability to identify variants of known attack methods. Although hybrid honeypots can solve some of these problems by deploying low-interaction honeypots and high-interaction honeypots, they cannot really be applied to real production scenarios because of their slow TCP connection switching speed and inability to efficiently identify encrypted malicious traffic. In this paper, we propose a new dynamic security defense system based on the combination of TCP_REPAIR-based dynamic honeypot selection architecture and a deep learning-based intelligent firewall. The system accurately distributes encrypted or non-encrypted attack traffic and its variants through the intelligent firewall. The normal traffic is sent to the actual system, and the marked malicious traffic dynamically selects honeypots to respond according to the attack process.The experimental result indicated that the system can select honeypots for targeted responses according to the actual network situation quickly and dynamically and covertly, effectively improving the utilization rate of honeypot clusters as well as the ability to decoy.
BibTeX
@article{tang2023new,
  author = {Jianxun Tang and Mingsong Chen and Haoyu Chen and Shenqi Zhao and Yu Huang},
  title = {{A new dynamic security defense system based on TCP\_REPAIR and deep learning}},
  journal = {J. Cloud Comput.},
  year = {2023},
  url = {https://openreview.net/forum?id=vnDF2SI8Sn},
}
Figure