Performance Evaluation System Based on Multi-Indicators for Signal Recognition

Qingming Li, Songlin Yang, Hu ChenIEEE Access 2023 · DOI

Introduction

In the modern era, the electromagnetic spectrum has become increasingly congested with a myriad of radiation sources. Identifying specific signals—often referred to as "fingerprinting"—is a critical task for security and spectrum management, particularly for ubiquitous devices like WiFi modules. While deep learning has significantly advanced the field of signal recognition, the research community has historically focused almost exclusively on recognition accuracy as the primary metric of success.

However, real-world deployment presents challenges that accuracy alone cannot capture. Models must operate on resource-constrained edge devices and maintain performance in high-interference environments. This paper addresses this gap by proposing a holistic, multi-dimensional evaluation framework that considers classification performance, computational complexity, and environmental robustness simultaneously.

The Multi-Indicator Evaluation Framework

The proposed system moves beyond simple accuracy by categorizing performance into three distinct pillars:

  1. Classification Performance: Evaluated through a suite of metrics including Confusion Matrices, Accuracy, Precision, Recall, and F1-score to provide a granular view of model reliability.
  2. Complexity: Quantified using Floating Point Operations (FLOPs), which serves as a proxy for the computational power and memory required for real-time inference on hardware.
  3. Robustness: Assessed by testing model stability under varying levels of Gaussian noise, specifically measuring performance across a Signal-to-Noise Ratio (SNR) range from -10 dB to 20 dB.
Overview of the signal recognition workflow and the integrated multi-indicator evaluation system.

Methodology and Experimental Setup

The authors benchmarked several prominent machine learning architectures against a specialized WiFi radiation source dataset. The models included standard Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), AlexNet, and ResNet, alongside the AdaBoost ensemble algorithm.

The study utilized a dataset comprising 100 distinct classes of 5GHz WiFi modules. Data consisted of I/Q (In-phase/Quadrature) signals collected in both laboratory and microwave darkroom settings, encompassing both Line-of-Sight (LOS) and Non-Line-of-Sight (NLOS) multipath conditions. Each of the 5,000 signals (4,000 training, 1,000 test) had a length of 5,000 samples.

Key Insights and Results

The Accuracy-Robustness Paradox

The experiments revealed a striking trade-off between a model's "intelligence" and its "resilience." ResNet achieved the highest accuracy in clean conditions (94.3%) but proved to be the most fragile when noise was introduced. In contrast, the basic DNN, which had the lowest baseline accuracy (77.4%), demonstrated the highest stability. At 0dB SNR, the DNN maintained 76.5% accuracy, while ResNet's performance plummeted to 11.2%.

Complexity vs. Efficiency

Interestingly, computational complexity did not always correlate with performance. The DNN exhibited the highest FLOPs but the worst recognition accuracy, suggesting that architectural depth and specialized feature extraction (as seen in CNNs) are far more efficient for signal processing than simply increasing parameter counts. AlexNet emerged as the most balanced model, offering high accuracy in low-noise conditions with significantly lower computational requirements (~14.4 million FLOPs) compared to ResNet (~166.6 million FLOPs).

The Risk of Over-Boosting

While AdaBoost is designed to strengthen weak learners, the study found it could actually degrade the performance of already strong models like AlexNet. This "over-boosting" effect likely stems from overfitting during the iterative training process, highlighting the need for careful algorithm selection based on the base learner's complexity.

Conclusion

This research demonstrates that a single-minded focus on accuracy is insufficient for practical signal recognition. By employing a multi-indicator evaluation system, developers can identify models that offer the best compromise between high-fidelity recognition, hardware efficiency, and environmental robustness. Future work will look to expand this framework to include adversarial security metrics and broader signal types beyond WiFi.

Summary updated

BibTeX
@article{li2023performance,
  author = {Qingming Li and Songlin Yang and Hu Chen},
  title = {{Performance Evaluation System Based on Multi-Indicators for Signal Recognition}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2022.3228641},
}
Figure