TL;DR
This paper investigates the challenges of using machine learning for side-channel attacks (SCAs) when the training data is imbalanced, a common issue in practice. The authors demonstrate that standard machine learning metrics like accuracy can be highly misleading in these scenarios, and that data sampling techniques, particularly SMOTE, can significantly improve attack performance by reducing the number of required measurements.
What Problem Does This Paper Address?
Side-channel attacks (SCAs) exploit physical leakages (like power consumption or electromagnetic radiation) from cryptographic devices to extract secret keys. Profiled SCAs, a common type, involve an attacker first profiling a device to learn its leakage patterns and then using this knowledge to attack a target device. Machine learning (ML) has emerged as a powerful tool for these attacks, often outperforming traditional template attacks, especially when profiling data is limited.
However, a significant challenge arises when the data used to train ML models is imbalanced. This occurs when certain leakage patterns (classes) are much more frequent than others, for instance, with Hamming weight or Hamming distance leakage models. In such cases, ML models can become biased towards the majority class, leading to high accuracy scores that don't reflect true attack success. This paper addresses the critical problem of how to effectively use ML for SCAs in the presence of imbalanced data and how to reliably assess the performance of these attacks, as standard ML metrics can be deceptive.
Methods
The authors explore various techniques to handle imbalanced data in the context of SCAs. These include:
- Data-level methods:
- Class Weight Balancing: Adjusting the weights of different classes during training to give more importance to minority classes.
- Random Undersampling: Randomly removing samples from majority classes to match the number of minority class samples.
- Random Oversampling: Randomly duplicating samples from minority classes.
- SMOTE (Synthetic Minority Over-sampling Technique): Generating synthetic minority class samples by interpolating between existing minority class samples and their nearest neighbors.
- SMOTE+ENN (Edited Nearest Neighbors): Combining SMOTE with a data cleaning technique (ENN) to remove noisy instances.
- Algorithm-level methods: While not the primary focus, the paper acknowledges these exist.
The paper also compares standard ML metrics (Accuracy, Precision, Recall, F1-score, G-mean, MCC, Kappa) with SCA-specific metrics like Success Rate (SR) and Guessing Entropy (GE). They investigate how well ML metrics can predict SCA performance and whether using SCA metrics directly in ML classifiers is feasible.
The ML classifiers used include Radial Kernel Support Vector Machines (SVM) and Random Forest (RF). They also explore deep learning models like Multilayer Perceptrons (MLP) and Convolutional Neural Networks (CNN), as well as traditional SCA techniques like Template Attack (TA) and its pooled version (TA p.), and Stochastic Attack (SA).
Datasets
The study uses three real-world datasets:
- DPAcontest v4: A masked AES software implementation dataset, characterized by high SNR. The authors adapt it to an unmasked scenario.
- AES_RD (Random Delay Countermeasure Dataset): An AES software implementation on an 8-bit microcontroller protected with a random delay countermeasure, resulting in low SNR.
- AES_HD (Unprotected AES-128 on FPGA): An unprotected AES-128 implementation on an FPGA, also with low SNR.
Key Insights and Ideas
- The "Curse" of Imbalanced Data: Highly imbalanced datasets, common in SCAs (e.g., Hamming weight models), can lead ML models to perform poorly in practice, even if their accuracy scores appear high.
- Misleading ML Metrics: Standard ML metrics like accuracy are often poor indicators of SCA success. A high accuracy can mask a low Success Rate or high Guessing Entropy, meaning the attacker might be making many correct predictions but still failing to recover the key efficiently.
- SMOTE is a Powerful Ally: Data sampling techniques, particularly SMOTE, are highly effective in mitigating the negative effects of imbalanced data in SCAs. SMOTE can significantly reduce the number of traces needed for a successful attack, sometimes by over 8 times.
- SCA Metrics are Crucial: For reliable evaluation of SCAs, especially in imbalanced scenarios, SCA-specific metrics like Success Rate and Guessing Entropy are essential. They provide a more accurate picture of an attacker's ability to recover the secret key.
- No One-Size-Fits-All: While SMOTE generally performs well, its effectiveness can vary depending on the dataset's noise level and the specific ML model used.
Key Results
- SMOTE's Impact: On the AES_RD dataset (low SNR, imbalanced), SMOTE reduced the number of traces needed for a successful attack by over 8 times compared to the imbalanced case.
- Accuracy vs. SR/GE: In the DPAcontest v4 dataset, accuracies for SVM and RF were similar (around 70%), but the Guessing Entropy differed significantly, highlighting the inadequacy of accuracy for performance assessment.
- AES_RD Difficulty: This dataset proved to be the most challenging. Even with SMOTE, reaching attack thresholds required a substantial number of traces, but SMOTE still offered significant improvements over imbalanced data.
- MLP and CNN Performance: Deep learning models (MLP and CNN) also benefited from SMOTE, though their performance varied. In some cases, SMOTE even slightly degraded CNN performance.
- Template Attack Behavior: Traditional template attacks (TA and TA pooled) were less affected by imbalanced data compared to ML models, but SMOTE could sometimes improve their performance, especially with smaller training sets.
Limitations and Future Work
- ML Metrics Limitations: The paper strongly emphasizes that standard ML metrics are not reliable indicators of SCA performance, especially with imbalanced data. They suggest that the community needs to be cautious when interpreting results based solely on these metrics.
- Designing New ML Metrics: The authors acknowledge that designing a new ML metric that directly incorporates SCA metrics (like GE/SR) into the classifier's training process is computationally expensive and complex, posing significant implementation challenges.
- Future Work: The authors suggest exploring how to design ML metrics that better reflect SCA behavior, potentially by incorporating SCA metrics directly into the ML process, despite the computational hurdles. They also hint at investigating the impact of countermeasures on imbalanced data scenarios.
Summary updated