To Overfit, Or Not to Overfit: Improving the Performance of Deep Learning-based SCA

Azade Rezaeezade, Guilherme Perin, Stjepan Picek

TL;DR

This paper challenges the common assumption that more data always leads to better generalization in deep learning-based side-channel analysis (SCA). The authors demonstrate that overfitting is often a consequence of suboptimal hyperparameter choices rather than insufficient data, and propose focusing on hyperparameter tuning and regularization for improved SCA model performance.

What Problem Does This Paper Address?

Side-channel analysis (SCA) is a crucial technique for evaluating the security of cryptographic devices by analyzing unintended information leakage. Profiling attacks, a type of SCA, use a "profiling model" trained on a device identical to the target to recover secret information. Deep learning models have shown promise in SCA due to their ability to learn complex patterns without manual feature engineering. However, a significant challenge with deep learning is hyperparameter tuning, which can lead to models that overfit the training data, resulting in poor generalization to unseen data. This overfitting can give a false sense of security. The paper addresses the question of whether simply increasing the amount of profiling data is the solution to this overfitting problem, or if other factors are at play.

Methods

The authors investigate the impact of profiling set size on the generalization performance of deep learning models in SCA. They employ a systematic approach involving:

  • Hyperparameter Search: Generating a large number of MLP and CNN model combinations with varying hyperparameters.
  • Model Selection: Identifying "good" models that can recover the correct key with a reasonable number of attack traces (guessing entropy of 1).
  • Re-training with Increased Data: Re-training these selected models with progressively larger profiling datasets.
  • Analysis of Regimes: Categorizing models into under-parameterized, critically-parameterized, and over-parameterized regimes based on their capacity and training data size, drawing parallels to the "deep double descent" phenomenon.
  • Investigating Regularization: Exploring the effect of dropout regularization on model performance.
  • Hyperparameter Tweaking: Examining how small changes in hyperparameters (e.g., learning rate, epochs, activation functions) can influence model performance.

Datasets

The paper utilizes two publicly available datasets:

  • ASCAD Random Keys: A dataset of side-channel measurements from an AES-128 software implementation protected by Boolean masking. It contains 200,000 traces for profiling and 100,000 for attack/validation. The features are trimmed to 1,400.
  • AES_HD: A dataset of power side-channel measurements from an unprotected AES-128 implementation on an FPGA. It contains 500,000 traces, split into a profiling set of 400,000 and 100,000 for attack/validation. Each measurement has 1,250 features.

Key Insights and Ideas

  • More Data Isn't Always Better: The core insight is that increasing the profiling dataset size does not consistently improve generalization in deep learning-based SCA. In many cases, performance either plateaus or even degrades.
  • Hyperparameters are Key: Overfitting and poor generalization are strongly linked to suboptimal hyperparameter choices, which can lead to "assumption errors" about the leakage model.
  • Regime Matters: The performance of deep learning models in SCA is highly dependent on the regime they operate in (under-parameterized, critically-parameterized, or over-parameterized), which is influenced by the model's capacity relative to the profiling data size.
  • Under-parameterized is Often Preferred: Contrary to some general machine learning trends, under-parameterized models often exhibit better generalization in SCA, especially for noisy datasets like AES-HD.
  • Deep Double Descent in SCA: The paper observes phenomena related to deep double descent, where increasing model complexity can initially hurt performance before improving again, but this behavior is complex and not always predictable in SCA.
  • Regularization and Tweaking Help: Small adjustments to hyperparameters and the use of regularization techniques like dropout can sometimes mitigate overfitting and improve generalization, even when simply adding more data does not.

Key Results

  • The study generated and evaluated 3,000 different MLP and CNN models across two datasets and leakage models.
  • Experiments showed that for many models, increasing profiling traces from 50,000 to 200,000 (ASCAD) or 400,000 (AES_HD) did not lead to a significant decrease in the required attack traces to achieve a guessing entropy of 1.
  • In some instances, increasing profiling data actually increased the number of attack traces needed, indicating a loss of generalization.
  • Models in the under-parameterized regime generally performed better than those in the critical or over-parameterized regimes.
  • Small changes in hyperparameters (e.g., number of epochs, learning rate, activation function) were shown to improve the performance of models that were initially underperforming or unable to converge.

Limitations and Future Work

  • Dataset Dependence: While the study used two distinct datasets, further investigation across a wider range of datasets and leakage models would be beneficial.
  • Hyperparameter Space: The search space for hyperparameters, while large, is finite. The authors acknowledge that finding the absolute optimal combination is challenging.
  • Systematic Approach: The paper suggests a need for a more systematic approach to hyperparameter tuning for SCA.
  • Regularization Evaluation: The authors note that while regularization shows promise, there's a lack of systematic evaluations of its relevance and effectiveness in SCA. Future work could focus on developing a comprehensive understanding of regularization techniques for SCA.

Summary updated

BibTeX
@article{rezaeezade2022overfit,
  author = {Azade Rezaeezade and Guilherme Perin and Stjepan Picek},
  title = {{To Overfit, Or Not to Overfit: Improving the Performance of Deep Learning-based SCA}},
  journal = {IACR Cryptol. ePrint Arch.},
  year = {2022},
}

Loading related content...

Figure