A Comparison of Weight Initializers in Deep Learning-Based Side-Channel Analysis

Huimin Li, Marina Krcek, Guilherme PerinIACR ePrint 2020 · OpenReview

TL;DR

This paper investigates how different weight initialization strategies impact the performance of deep learning models in side-channel analysis (SCA). The authors find that the choice of initializer significantly affects attack success, especially on more challenging datasets, and that its performance is strongly linked to the activation function used.

What Problem Does This Paper Address?

Side-channel analysis (SCA) is a powerful class of attacks that exploits physical leakage (like power consumption or electromagnetic radiation) from a device to infer secret information. Profiled SCA, which uses a "profile" of a device's leakage, has seen significant advancements with deep learning techniques, particularly Convolutional Neural Networks (CNNs). However, training these deep learning models involves numerous hyperparameters, and their optimal selection remains a challenge. This paper focuses on one such critical hyperparameter: the weight initializer, which sets the initial values of the neural network's weights. The authors aim to understand how different initialization strategies affect the success of deep learning-based SCA and how this choice interacts with other hyperparameters and datasets.

Methods

The researchers conducted extensive experiments using 11 different weight initializers available in the Keras library. They evaluated these initializers across two distinct CNN architectures (referred to as "Methodology" and "Noise") and three publicly available datasets (DPAv4, AES_RD, and ASCAD). For each dataset and architecture combination, they tested two leakage models: Identity (ID) and Hamming Weight (HW). The primary metric for evaluating attack performance was Guessing Entropy (GE), which measures how quickly the correct key byte can be identified. They also analyzed the stability of results across multiple training runs and observed the evolution of weights and biases during training.

Datasets

The paper utilizes three publicly available datasets for side-channel analysis:

  • DPA contest v4 (DPAv4): Derived from a masked AES software implementation, allowing conversion to an unprotected scenario.
  • AES_RD: Obtained from an 8-bit AVR microcontroller with a random delay countermeasure.
  • ASCAD: A dataset from a masked AES-128 implementation on an 8-bit AVR microcontroller.

Key Insights and Ideas

  • Initializer Choice Matters, Especially for Difficult Attacks: While many initializers perform similarly on "easy" datasets, the choice becomes crucial for more challenging SCA scenarios.
  • Activation Function is Key: The paper strongly suggests that the choice of weight initializer is more dependent on the activation function than other hyperparameters like learning rate or batch size. This is because many initializers are designed with specific assumptions about activation functions (e.g., Glorot assumes linearity, while He is designed for ReLU).
  • Training is Not Monolithic: Significant differences were observed in individual training phases, highlighting the need for multiple training runs and statistical analysis rather than relying on a single attack outcome.
  • Layer-Specific Weight Evolution: Weight and bias changes during training are most pronounced in Convolutional and Batch Normalization layers, while fully-connected (dense) layers tend to remain relatively stable.
  • Architecture Differences: The "Methodology" architecture showed more significant weight and bias changes during training compared to the "Noise" architecture, which appeared more "robust" and required less weight adjustment for strong performance.

Key Results

The study found that:

  • Random initializers (like RandomUniform and RandomNormal) often performed well, especially on the DPAv4 dataset.
  • Heuristic-based initializers (he_normal, lecun_normal, glorot_uniform) generally showed good performance, particularly on more complex datasets like AES_RD and ASCAD. he_normal and lecun_normal were frequently identified as the best performers.
  • Zeros and Ones initialization consistently resulted in performance no better than random guessing, confirming they are insufficient for these tasks.
  • The best initializer varied across datasets and architectures, indicating no single "one-size-fits-all" solution. For instance, on ASCAD with the Methodology architecture, he_normal was found to be superior.
  • The paper also explored the impact of varying other hyperparameters (filters, kernel size, neurons, layers, learning rate, mini-batch size) on the best-performing initializers. They found that while filter count and activation functions had a noticeable impact, variations in kernel size, number of layers/neurons, learning rate, and mini-batch size had less significant effects on the final GE.

Limitations and Future Work

The authors acknowledge that mathematical explanations of weight initialization strategies were beyond the scope of this work. They suggest two main directions for future research:

  • Further exploring the interplay between weight initializers and activation functions: This is a critical area, as their results strongly indicate a significant dependency.
  • Investigating unsupervised pre-training setups: Autoencoders could potentially assign weights in an unsupervised manner, guiding learning towards better generalization.

Summary updated

Abstract

The usage of deep learning in profiled side-channel analysis requires a careful selection of neural network hyperparameters. In recent publications, different network architectures have been presented as efficient profiled methods against protected AES implementations. Indeed, completely different convolutional neural network models have presented similar performance against public side-channel traces databases. In this work, we analyze how weight initializers’ choice influences deep neural networks’ performance in the profiled side-channel analysis. Our results show that different weight initializers provide radically different behavior. We observe that even high-performing initializers can reach significantly different performance when conducting multiple training phases. Finally, we found that this hyperparameter is more dependent on the choice of dataset than other, commonly examined, hyperparameters. When evaluating the connections with other hyperparameters, the biggest connection is observed with activation functions.
BibTeX
@article{li2020comparison,
  author = {Huimin Li and Marina Krcek and Guilherme Perin},
  title = {{A Comparison of Weight Initializers in Deep Learning-Based Side-Channel Analysis}},
  journal = {IACR Cryptol. ePrint Arch.},
  year = {2020},
  url = {https://openreview.net/forum?id=guNcQCbMrh},
}
Figure