Reinforcement Learning-based Design of Side-channel Countermeasures

Jorai Rijsdijk, Lichao Wu, Guilherme Perin, Stjepan Picek

Abstract

Side-channel analysis (SCA) poses a significant threat to cryptographic hardware, with deep learning (DL) techniques increasingly capable of bypassing traditional defenses. Designing effective countermeasures requires balancing security against implementation costs, a process that has historically been manual and heuristic. This paper introduces a novel framework that utilizes reinforcement learning (RL) to automate the selection of optimal "hiding" countermeasures. By modeling the design space as a Markov Decision Process, the framework identifies low-cost protection schemes—specifically those leveraging temporal desynchronization—that successfully thwart state-of-the-art deep learning attacks.

Introduction

Side-channel analysis exploits physical leakages, such as power consumption or electromagnetic emissions, to extract secret keys from cryptographic devices. While "masking" (randomizing intermediate values) has long been a standard defense, the advent of deep learning has shifted the landscape. Modern convolutional neural networks (CNNs) can often bypass masking with minimal pre-processing, necessitating the use of "hiding" techniques that reduce the signal-to-noise ratio or disrupt the temporal alignment of leakages.

Designers face a complex optimization problem: which combination of noise injection, clock jitter, or random delays provides the best security for a given performance budget? This paper addresses the lack of systematic methods for this selection by proposing an automated framework based on Tabular Q-Learning.

Methodology

The proposed framework treats the selection of countermeasures as a search problem within a reinforcement learning environment. The system utilizes a Tabular Q-Learning approach (based on MetaQNN) to navigate the trade-offs between security and overhead.

The RL Environment

The selection process is modeled as a Markov Decision Process (MDP) where an RL agent chooses from a menu of four primary hiding countermeasures:

  1. Desynchronization: Shifting traces in time to misalign leakage points.
  2. Uniform Noise: Adding random values to the signal amplitude.
  3. Clock Jitter: Introducing instability into the clock signal to create non-linear temporal distortions.
  4. Random Delay Interrupts (RDIs): Inserting dummy instructions (e.g., nop) at random intervals to break the temporal consistency of the execution.

Reward Function and Attack Loop

The agent's "brain" is a reward function designed to maximize security while minimizing cost. A high reward is granted if the selected countermeasure set:

  • Increases the Guessing Entropy (GE), ensuring the secret key remains hidden.
  • Prevents a successful attack within a fixed limit of 2,000 traces.
  • Stays within a predefined cost budget ($c_{max}$).

To evaluate a proposed configuration, the framework simulates the countermeasures on a dataset and subjects them to an attack loop using optimized CNNs. The agent learns from the success or failure of these attacks, iteratively refining its strategy.

Experimental Setup

The framework was evaluated using the ASCAD database, the industry standard for profiling side-channel analysis. Two specific datasets were used:

  • ASCAD Fixed Key: 50,000 traces from an 8-bit AVR microcontroller running masked AES-128 with a constant key.
  • ASCAD Random Keys: 300,000 traces with varying keys, requiring the DL model to learn more generalized leakage representations.

The study considered both Hamming Weight (HW) and Identity (ID) leakage models to ensure the robustness of the discovered defenses.

Results and Discussion

The Superiority of Temporal Defenses

A primary insight from the study is that time-based countermeasures are significantly more effective than amplitude-based noise. While DL models are adept at filtering out uniform noise, they struggle when the features they seek are shifted or delayed unpredictably. Specifically, Random Delay Interrupts (RDIs) emerged as the most valuable player (MVP), consistently appearing in top-performing, low-cost sets. Even at low probabilities, RDIs drastically increased the complexity of the profiling task for the CNNs.

Efficiency and Convergence

The RL agent demonstrated clear learning progress over thousands of iterations (approximately 200–300 GPU hours). By incorporating implementation costs into the reward function, the framework prevented the agent from simply "maxing out" every defense. Instead, the agent converged on clever, lightweight combinations—such as a specific level of desynchronization—that provided sufficient security within a cost budget as low as 5 units.

Generalization Challenges

The research noted that attacks on the "Random Keys" dataset required more expensive countermeasures to defeat. This suggests that as DL models are forced to learn more general representations of leakage, they become more resilient to simple hiding techniques, highlighting the need for the adaptive, automated design approach provided by this framework.

Conclusion

This work demonstrates that reinforcement learning can effectively automate the design of side-channel countermeasures, identifying configurations that are both secure and efficient. The findings underscore the critical importance of temporal desynchronization in modern hardware security. Future research may expand this framework to optimize masking logic itself or to adaptively respond to new, specifically tuned DL attack architectures.

Summary updated

BibTeX
@article{rijsdijk2021reinforcement,
  author = {Jorai Rijsdijk and Lichao Wu and Guilherme Perin and Stjepan Picek},
  title = {{Reinforcement Learning-based Design of Side-channel Countermeasures}},
  journal = {IACR Cryptol. ePrint Arch.},
  year = {2021},
  url = {https://openreview.net/forum?id=G9ZrDAuqAX},
}
Figure