Online Performance Evaluation of Deep Learning Networks for Side-Channel Analysis.

Damien Robissout, Gabriel Zaid, Brice Colombier, Lilian Bossuet, Amaury Habrard

TL;DR

This paper introduces a novel metric, $\Delta_{train,val}$, to evaluate the generalization performance of deep learning models in side-channel analysis. By comparing the number of traces required for a model to achieve a 90% success rate on both training and validation sets, this metric effectively detects underfitting and overfitting, enabling early stopping and improving model efficiency.

What Problem Does This Paper Address?

Side-channel attacks exploit physical leakage (like power consumption) from cryptographic devices to extract secret keys. Deep learning, particularly Convolutional Neural Networks (CNNs), has shown promise in these attacks, offering robustness against countermeasures. However, a critical challenge is evaluating how well these deep learning models generalize. Traditional metrics like accuracy are insufficient because they don't account for the cumulative nature of side-channel analysis, where multiple traces are needed to infer the secret. Existing methods struggle to pinpoint the optimal training point, leading to either underfitting (model too simple) or overfitting (model memorizes training data, failing on new data), both of which degrade performance and waste computational resources.

Methods

The core contribution is the introduction of $\Delta_{train,val}$, a new evaluation metric for deep learning in side-channel analysis. This metric is defined as the absolute difference between the number of traces needed to achieve a 90% d-th order success rate on the training set ($N_{train}$) and the validation set ($N_{val}$).

$\Delta_{train,val} = |N_{val} - N_{train}|$

By monitoring $\Delta_{train,val}$ during training, the authors propose:

  • Internal State Detection:
    • Underfitting: Characterized by high or undefined $\Delta_{train,val}$, indicating the model hasn't learned enough from the training data.
  • Good Trade-off: When $\Delta_{train,val}$ converges towards a minimal value ($N_{bias}$), signifying good generalization.
  • Overfitting: Indicated by $\Delta_{train,val}$ increasing significantly as $N_{train}$ approaches 1 and $N_{val}$ grows, showing the model memorizes training data at the expense of generalization.
  • Early Stopping: The metric allows for stopping training at the "best trade-off" point, preventing overfitting and saving computation time.

The authors also compare their metric to the Guessing Entropy Bias Variance Decomposition (GEBVD) [15], another method for analyzing model states.

Datasets

  • ASCAD database [13]: This is the primary dataset used for training and evaluation. It contains electromagnetic measurements from an 8-bit AVR ATMega8515 running an AES implementation secured against first-order side-channel attacks. It includes a training set of 50,000 traces and a test set of 10,000 traces. The leakage model used is $Y(k^*) = Sbox(p[3] \oplus k^*[3])$, focusing on the masked value of the third key byte in the first round.

Key Insights and Ideas

  • Success Rate is Key: Traditional accuracy is insufficient for side-channel analysis. The success rate (SR), which measures how often the correct key is among the top-d guesses, is a more appropriate metric.
  • Generalization is Paramount: The true value of a deep learning model in side-channel analysis lies in its ability to generalize to unseen traces. $\Delta_{train,val}$ directly quantifies this generalization.
  • Early Stopping Pays Off: By identifying the optimal training epoch using $\Delta_{train,val}$, significant improvements in both attack performance (fewer traces needed) and training time can be achieved.
  • $\Delta_{train,val}$ vs. GEBVD: The paper suggests $\Delta_{train,val}$ offers a more intuitive and direct insight into model state, especially when attacks are successful (high SR), where GEBVD can become less informative.
  • CCE Loss Advantage: The study indicates that the categorical cross-entropy (CCE) loss function is generally more suitable for deep learning in side-channel analysis compared to mean squared error (MSE), leading to better performance and reduced variance.

Key Results

  • Early Stopping Improvement: On the ASCAD database, using $\Delta_{train,val}$ for early stopping on the CNNbest architecture resulted in:
  • A 30.3% reduction in the number of traces needed to reach a 90% success rate.
  • A 33.3% reduction in training time (from 1 hour to 40 minutes).
  • CNN BV Performance: For the CNNBV architecture, $\Delta_{train,val}$ identified optimal training points that required fewer traces and less training time compared to previously recommended epochs in the literature. For example, with 0 convolutional blocks and CCE loss, the optimal epoch was 37, requiring 1300 traces, compared to 50 epochs and 1800 traces in prior work.
  • Overfitting Detection: The metric clearly showed overfitting occurring after the optimal training point, with $\Delta_{train,val}$ increasing as the model's performance on the training set continued to improve while validation performance stagnated or degraded.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while $\Delta_{train,val}$ helps identify the "best trade-off," it doesn't guarantee an "optimal" network, as hyperparameter tuning and architecture choices still play a significant role.
  • The paper notes that GEBVD can be less informative when guessing entropy reaches 0 or 1, whereas $\Delta_{train,val}$ can still provide insights.
  • Future Work:
    • Evaluating the impact of normalization and regularization techniques using $\Delta_{train,val}$.
  • Adapting $\Delta_{train,val}$ (or a derivative) as a loss function to directly optimize the network for side-channel analysis during training.

Summary updated

Abstract

Deep learning based side-channel analysis has seen a rise in popularity over the last few years. A lot of work is done to understand the inner workings of the neural networks used to perform the attacks and a lot is still left to do. However, finding a metric suitable for evaluating the capacity of the neural networks is an open problem that is discussed in many articles. We propose an answer to this problem by introducing an online evaluation metric dedicated to the context of side-channel analysis and use it to perform early stopping on existing convolutional neural networks found in the literature. This metric compares the performance of a network on the training set and on the validation set to detect underfitting and overfitting. Consequently, we improve the performance of the networks by finding their best training epoch and thus reduce the number of traces used by 30%. The training time is also reduced for most of the networks considered.
BibTeX
@inproceedings{robissout2020online,
  author = {Damien Robissout and Gabriel Zaid and Brice Colombier and Lilian Bossuet and Amaury Habrard},
  title = {{Online Performance Evaluation of Deep Learning Networks for Side-Channel Analysis.}},
  booktitle = {IACR Cryptology ePrint Archive},
  year = {2020},
  url = {https://openreview.net/forum?id=SgXy0b72a5Q},
}

Loading related content...

Figure