Enhanced Real-Time Threat Detection in 5G Networks: A Self-Attention RNN Autoencoder Approach for Spectral Intrusion Analysis

Mohammadreza Kouchaki, Minglong Zhang, Aly Sabri Abdalla, Guangchen Lan, Christopher G. Brinton, Vuk Marojevic

Abstract

In the rapidly evolving landscape of 5G technology, safeguarding Radio Frequency (RF) environments against sophisticated intrusions is paramount, especially in dynamic spectrum access and management. This paper presents an enhanced experimental model that integrates a self-attention mechanism with a Recurrent Neural Network (RNN)-based autoencoder for the detection of anomalous spectral activities in 5G networks at the waveform level. Our approach, grounded in time-series analysis, processes in-phase and quadrature (I/Q) samples to identify irregularities that could indicate potential jamming attacks. The model's architecture, augmented with a self-attention layer, extends the capabilities of RNN autoencoders, enabling a more nuanced understanding of temporal dependencies and contextual relationships within the RF spectrum. Utilizing a simulated 5G Radio Access Network (RAN) test-bed constructed with srsRAN 5G and Software Defined Radios (SDRs), we generated a comprehensive stream of data that reflects real-world RF spectrum conditions and attack scenarios. The model is trained to reconstruct standard signal behavior, establishing a normative baseline against which deviations, indicative of security threats, are identified. The proposed architecture is designed to balance between detection precision and computational efficiency, so the LSTM network, enriched with self-attention, continues to optimize for minimal execution latency and power consumption. Conducted on a real-world SDR-based testbed, our results demonstrate the model's improved performance and accuracy in threat detection.

Introduction

As 5G networks transition toward dynamic spectrum access and open architectures like O-RAN, they face increased vulnerability to Radio Frequency (RF) interference and jamming. Traditional Intrusion Detection Systems (IDS) often rely on static signatures, which are ineffective against adaptive "intelligent" jamming or Advanced Persistent Threats (APTs). 5G signals, characterized by complex time-series data, require models capable of understanding long-term temporal dependencies at the waveform level to detect anomalies amidst high-density network noise.

Proposed Architecture

The proposed system utilizes a Self-Attention RNN Autoencoder designed for unsupervised anomaly detection. The architecture consists of two primary components:

  1. RNN Autoencoder: An encoder compresses raw In-phase and Quadrature (I/Q) samples into a lower-dimensional latent representation, while a decoder attempts to reconstruct the original signal.
  2. Self-Attention Mechanism: Integrated into the encoder, a multi-head self-attention layer allows the model to dynamically weigh different segments of the input sequence. This enhancement enables the capture of complex contextual relationships within the RF spectrum that standard Long Short-Term Memory (LSTM) networks might overlook.
Self-Attention RNN Autoencoder architecture for I/Q sample reconstruction.

The system identifies intrusions by calculating the Mean Squared Error (MSE) between the input and its reconstruction. If the error exceeds a predefined threshold, a "Spectral Intrusion" is flagged. This unsupervised approach allows for the detection of "zero-day" attacks without prior knowledge of specific threat signatures.

Experimental Setup and Datasets

To validate the model, a custom 5G Radio Access Network (RAN) testbed was developed. The setup utilized srsRAN 5G for the gNodeB and User Equipment (UE), and open5GS for the core network. Hardware implementation involved Software Defined Radios (USRPs) for live RF transmission. A dataset of approximately 300 million I/Q samples was collected. To simulate real-world threats, static and dynamic jamming signals were synthetically infused into the live 5G stream using GNURadio.

Results and Discussion

Experimental results indicate high reconstruction fidelity under normal operating conditions, confirming the model's ability to learn the unique "fingerprint" of 5G signals. Upon the introduction of jamming, the reconstruction error (MSE) exhibited significant spikes, providing a clear demarcation between normal traffic and intrusion events.

Comparison of original and reconstructed I/Q data under normal and intrusion conditions.

The architecture successfully balances detection precision with computational efficiency. By processing raw I/Q samples at the physical layer, the system achieves faster detection compared to traditional software-based IDS that rely on higher-level network logs. Visual analysis of the reconstruction error shows that the model effectively isolates anomalous sequences (such as Sequence 262) where the reconstructed signal fails to match the original due to the presence of jamming.

Conclusion

This research demonstrates that combining self-attention with RNN autoencoders significantly enhances real-time spectral intrusion detection in 5G environments. Future work will focus on expanding the dataset to include more diverse environmental conditions and investigating adaptive thresholding to mitigate false positives. Additionally, the resilience of the model against more sophisticated, adversarial tactics remains a key area for exploration.

Summary updated

Abstract

In the rapidly evolving landscape of 5G technology, safeguarding Radio Frequency (RF) environments against sophisticated intrusions is paramount, especially in dynamic spectrum access and management. This paper presents an enhanced experimental model that integrates a self-attention mechanism with a Recurrent Neural Network (RNN)-based autoencoder for the detection of anomalous spectral activities in 5G networks at the waveform level. Our approach, grounded in time-series analysis, processes in-phase and quadrature (I/Q) samples to identify irregularities that could indicate potential jamming attacks. The model's architecture, augmented with a self-attention layer, extends the capabilities of RNN autoencoders, enabling a more nuanced understanding of temporal dependencies and contextual relationships within the RF spectrum. Utilizing a simulated 5G Radio Access Network (RAN) test-bed constructed with srsRAN 5G and Software Defined Radios (SDRs), we generated a comprehensive stream of data that reflects real-world RF spectrum conditions and attack scenarios. The model is trained to reconstruct standard signal behavior, establishing a normative baseline against which deviations, indicative of security threats, are identified. The proposed architecture is designed to balance between detection precision and computational efficiency, so the LSTM network, enriched with self-attention, continues to optimize for minimal execution latency and power consumption. Conducted on a real-world SDR-based testbed, our results demonstrate the model's improved performance and accuracy in threat detection. Keywords: self-attention, real-time intrusion detection, RNN autoencoder, Transformer architecture, LSTM, time series anomaly detection, 5G Security, spectrum access security.
BibTeX
@inproceedings{kouchaki2024enhanced,
  author = {Mohammadreza Kouchaki and Minglong Zhang and Aly Sabri Abdalla and Guangchen Lan and Christopher G. Brinton and Vuk Marojevic},
  title = {{Enhanced Real-Time Threat Detection in 5G Networks: A Self-Attention RNN Autoencoder Approach for Spectral Intrusion Analysis}},
  booktitle = {WiOpt},
  year = {2024},
  url = {https://openreview.net/forum?id=jjHDOVu84g},
}
Figure