When researchers compare attack methods in deep learning side-channel analysis (DL-SCA) — using the physical power or electromagnetic leakage of a chip running AES to recover its secret key — the published leaderboard may be an artifact of the measurement protocol rather than the method. Three separate findings point at this. The training objective (categorical cross-entropy) does not match what the attacker actually needs, which is a good ranking of key candidates after aggregating many traces Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis. The standard way of summarizing experiments — the arithmetic mean of Guessing Entropy, a score of how close the attack is to the true key — is skewed by unlucky training runs, and the median is the robust choice On the Evaluation of Deep Learning-based Side-channel Analysis. And in other fields, when every method gets the same tuning budget and the test set is kept out of model selection, supposedly superior modern methods routinely lose to decade-old baselines BARS: Towards Open Benchmarking for Recommender Systems, Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline.
This note proposes to test, in one controlled study, whether these three problems interact. The core hypothesis is that in DL-SCA the benchmarking-inconsistency problem is coupled to the metric-misalignment problem: which method wins depends not just on tuning budget but on the choice of selection metric and aggregation statistic, because the evaluation quantity (Guessing Entropy over many traces) is a fragile, high-variance functional of the model rather than a clean pointwise accuracy. If true, many published comparisons in the field are unreliable in a stronger sense than the standardization literature has shown, and fixing it requires locking down all three axes at once.
Core Claim
The claim to be tested: method rankings in DL-SCA are not invariant to protocol choices. Specifically, whether ranking-loss training beats cross-entropy, and whether deep networks beat classical profiling attacks, can flip depending on (a) the statistic used to aggregate Guessing Entropy across training runs (mean vs. median), (b) the metric used to select checkpoints (validation loss vs. fast, domain-aligned Guessing Entropy The Need for Speed: A Fast Guessing Entropy Calculation for Deep Learning-based SCA), and (c) whether model selection touches the final attack traces. No existing work isolates these factors. Wu, Perin and Picek varied only the aggregation statistic, and only for networks trained with cross-entropy On the Evaluation of Deep Learning-based Side-channel Analysis] — they never asked whether the statistic changes the ranking of methods. Zaid et al. showed ranking loss beats cross-entropy, but on fragmented internal pipelines with no budget equalization Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis. AISY standardizes the pipeline but hard-codes the flawed arithmetic-mean evaluation the community already knows is biased AISY - Deep Learning-based Framework for Side-channel Analysis, On the Evaluation of Deep Learning-based Side-channel Analysis. BARS and the point-cloud study equalize budgets, but in domains where the evaluation metric is pointwise accuracy or simple ranking recall — precisely the setting where protocol noise is lowest; they never touch an aggregate, multi-trace metric like Guessing Entropy BARS: Towards Open Benchmarking for Recommender Systems, Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline. The proposal differs from all of these by making protocol choices themselves the experimental factors, crossed factorially with the methods being compared.
Research Program
The program has three components, all released as a single locked-down execution configuration so that every number below is reproducible by construction.
First, a protocol engine: fixed data splits with a strict three-way partition (training / validation for checkpoint and hyperparameter selection / held-out attack traces touched exactly once), selection performed only on the validation partition using either validation loss or Fast Guessing Entropy (a sub-sampled Guessing Entropy computation that is up to 16× cheaper than the empirical one The Need for Speed: A Fast Guessing Entropy Calculation for Deep Learning-based SCA), and final reporting via the median Guessing Entropy across runs, with the run count set by the observed stabilization threshold of roughly 40 experiments On the Evaluation of Deep Learning-based Side-channel Analysis. The reported headline number is the trace count to key recovery:
$$T^* \;=\; \min\{\, t : \mathrm{median}_{i} \, \mathrm{GE}_i(t) = 0 \,\},$$
where $\mathrm{GE}_i(t)$ is the Guessing Entropy after aggregating $t$ attack traces in the $i$-th independent training run. The median makes $T^*$ insensitive to non-converging seeds, which is exactly the distortion documented for the arithmetic mean On the Evaluation of Deep Learning-based Side-channel Analysis.
Second, a budget equalization rule modeled on BARS BARS: Towards Open Benchmarking for Recommender Systems but adapted to a domain where methods differ in per-epoch cost. Ranking-loss training carries a reported 4× per-epoch overhead Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis, so equalizing trials would favor it and equalizing epochs would penalize it. We equalize wall-clock GPU budget per method (calibrated against the roughly 10 hours AutoSCA reports for a single tuning campaign on a GTX 1080 Ti I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis]), and report both budget conventions so the reader can see which conclusions are budget-invariant.
Third, a factorial meta-benchmark crossing method with protocol. Methods span the field's real contenders: multi-layer perceptrons and convolutional networks trained with cross-entropy (as in AISY AISY - Deep Learning-based Framework for Side-channel Analysis], with and without SMOTE class balancing, which cut a Random Forest's attack cost on AES_RD from 13,500 traces to 1,600 The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel Evaluations]), ranking-loss training with its steepness parameter $\alpha$ inside the search space (closing the gap AutoSCA leaves open — its Bayesian search tunes architectures but never the loss I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis]), and classical profiling attacks (template attacks, Random Forest with SMOTE) given the same wall-clock search over their own hyperparameters, which no SCA framework has done. Protocol factors are aggregation statistic (mean vs. median), selection metric (validation loss vs. FGE), and selection leakage (validation-only vs. the common but flawed practice of stopping on the attack set, the SCA analogue of the test-set epoch selection exposed in point-cloud work Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline]). The deliverable is a ranking-flip analysis: for each pair of methods, count how many protocol configurations reverse their order, and quantify the variance of $T^*$ attributable to protocol versus method.
Baselines
The naive reference point is the field's default practice: cross-entropy training with early stopping, arithmetic-mean evaluation over 100 runs, and selection on whatever set is convenient — this is essentially what AISY automates AISY - Deep Learning-based Framework for Side-channel Analysis], so we run it as-is, unmodified, to establish how far the incumbent protocol's numbers sit from the leakage-free, median-based ones on identical data. Beating it means showing that its reported $T^*$ under mean aggregation and test-set selection is systematically optimistic or pessimistic relative to the strict protocol, and that the direction and size of the bias are stable across datasets.
The strongest existing competitor is AutoSCA, the Bayesian-optimization tuning pipeline built on Keras Tuner that selects models by domain metrics (key rank) rather than loss I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis]. It reports key recovery on ASCAD fixed-key with 447 traces for its best multi-layer perceptron and 136 traces for its best convolutional network, against 1,124 traces for prior hand-designed models. We rerun it inside the protocol engine: same ASCAD_F splits, same held-out attack traces, same median-based reporting, same wall-clock budget we grant every other method. Beating it means a method that reaches median Guessing Entropy of zero with strictly fewer traces than 136 on ASCAD_F under the identical budget and reporting rule, or matching it at materially lower cost. This is the comparison that matters most, because AutoSCA is the only prior system that already couples automated tuning with domain-aligned selection — our claim is that even it is protocol-sensitive, since its numbers were produced under aggregation and selection conventions we show to be biased.
The key ablation isolates the protocol ingredient from the method ingredient: train one fixed architecture (a convolutional network of the kind AISY and AutoSCA both use) with cross-entropy only, then evaluate it under all protocol configurations — mean vs. median aggregation, validation-loss vs. FGE checkpoint selection, leakage-free vs. leaky stopping. Any spread in $T^*$ across these configurations, with the model held fixed, is pure protocol effect and directly measures how much of a published gap between two methods could be manufactured by protocol alone. This ablation is what distinguishes the proposal from a mere re-tuning exercise, and it is the experiment a reviewer would demand before believing any ranking-flip claim.
Why not just…?
Why not just use cross-entropy with early stopping on Guessing Entropy, as AISY already does? This is the cheapest incumbent and must be taken seriously. The evidence against it being sufficient is that cross-entropy optimizes only a lower bound on the information the attack needs, and on the low-SNR AES_HD dataset ranking-loss training reached key recovery with 20,000 profiling traces versus 24,870 for cross-entropy, converging up to 23% faster and staying robust under injected Gaussian noise where cross-entropy models failed Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis]. But the honest answer is that this comparison itself was run on unstandardized pipelines, so the gap may shrink or vanish under budget equalization — and testing exactly that is Experiment E2 of this proposal, not an argument outside it. If the gap survives equal wall-clock budgets despite ranking loss's 4× per-epoch cost, cross-entropy-plus-early-stopping is beaten on its own terms.
Why not just balance the data with SMOTE instead of touching losses or protocols? SMOTE is a real, cheap fix for the class-imbalance collapse that otherwise destroys distinguishing power, and it delivered an 8.4× improvement for a Random Forest on AES_RD The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel Evaluations]. However, the same paper found deep networks remain vulnerable to imbalance even with balancing, and — more important for this proposal — balancing addresses the training-side symptom, not the evaluation-side distortion, since Guessing Entropy can improve while classification accuracy stays flat. SMOTE is therefore included as a method factor in the factorial design rather than treated as a rival solution; if it closes the gap to ranking loss under a fair protocol, that is a finding the study is designed to detect and report.
Why not just build another framework on top of AISY instead of a new protocol engine? Building on AISY would save engineering effort, and its modular design makes it the natural host. The reason not to is that its defaults embed the two flaws this study targets: arithmetic-mean aggregation over 100 runs AISY - Deep Learning-based Framework for Side-channel Analysis], which the evaluation study showed to be outlier-skewed On the Evaluation of Deep Learning-based Side-channel Analysis], and its rigid ASCAD-shaped .h5 input format, which blocks the classical baselines and non-ASCAD datasets the equal-budget comparison needs. The pragmatic resolution is to implement the protocol engine as a thin configuration layer that reuses AISY's model definitions and data loaders where possible; the contribution is the protocol, not the codebase. If AISY's maintainers adopt the configuration, the framework question dissolves.
Why not just use random hyperparameter search instead of Bayesian optimization inside the tuning loop? Random search is simpler and parallelizable, and on easy datasets it is genuinely sufficient — that is the "easy dataset trap" AutoSCA itself identifies I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis]. But in high-noise settings, Bayesian-optimization-discovered models converged to a successful attack where random-search models failed, and its CNN reached 136 traces on ASCAD_F where prior manual designs needed 1,124 I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis]. More to the point, the tuning strategy is held constant across all methods in this proposal, so it cannot confound the comparison; random search enters only as a sanity check that our conclusions are not artifacts of one search algorithm.
Proposed Experiments
Experiment E1 replicates published numbers under the strict protocol. On ASCAD fixed-key ASCAD is used across AISY, AutoSCA, FGE and the tutorial papers, we reproduce the AutoSCA pipeline and report its 136-trace CNN result and 447-trace MLP result under median aggregation and validation-only selection; on AES_HD we reproduce the ranking-loss-versus-cross-entropy comparison (20,000 vs. 24,870 profiling traces to key recovery Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis]). The measure of interest is the delta between published and strict-protocol numbers. For CHES CTF 2018, no reported key-recovery trace count exists in the file, so we state plainly: no reported result found on Lacuna for CHES CTF trace counts; the dataset serves only as a third replication target whose numbers we generate ourselves.
Experiment E2 is the factorial protocol-sensitivity study. On ASCAD_F, ASCAD_R (random keys), and AES_HD, we cross the method set with the three protocol factors and measure $T^*$ plus the fraction of method pairs whose order flips across protocol configurations. The secondary check replicates the ~40-run stabilization finding of the evaluation study On the Evaluation of Deep Learning-based Side-channel Analysis] — this time for ranking-loss and balanced training, which that paper never tested — and asks whether the stabilization threshold itself is loss-dependent; if ranking-loss training has heavier-tailed convergence, the case for median aggregation strengthens, and if not, the mean is rehabilitated. Both outcomes are informative.
Experiment E3 is the equal-budget classical-versus-deep comparison, run on AES_RD (the low-SNR, random-delay dataset where SMOTE-balanced Random Forest needed 1,600 traces The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel Evaluations]) and AES_HD, the two hardest datasets in the file, deliberately avoiding the easy-dataset trap that AutoSCA warns can mask real differences I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis]. Template attacks and balanced Random Forests get the same wall-clock search budget over their hyperparameters as the networks. This is the SCA analogue of the BARS finding that decade-old baselines matched modern graph networks under equal grid search BARS: Towards Open Benchmarking for Recommender Systems], and the outcome is genuinely open: nobody has run it.
Decisive Checks
The study succeeds if it produces a calibrated answer to one question: how much of a reported trace-count improvement between two DL-SCA methods can be produced by protocol choices alone? The fixed-architecture ablation gives the protocol effect in isolation; the factorial gives the interaction with methods; E1 and E3 anchor everything against the field's actual published numbers. If rankings turn out to be protocol-invariant, that is a negative result worth publishing — it certifies the field's comparisons and rehabilitates the mean — but the priors from the evaluation study's outlier analysis On the Evaluation of Deep Learning-based Side-channel Analysis] and the budget-equalization surprises in recommendation and point clouds BARS: Towards Open Benchmarking for Recommender Systems, Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline] both point the other way. Either way, the released configuration makes every subsequent claim in the field checkable against a locked protocol, which is the lasting contribution.
References
- Ranking Loss: Maximizing the Success Rate in Deep Learning Side-Channel Analysis
- On the Evaluation of Deep Learning-Based Side-Channel Analysis
- BARS: Towards Open Benchmarking for Recommender Systems
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- The Need for Speed: A Fast Guessing Entropy Calculation for Deep Learning-Based SCA
- AISY - Deep Learning-based Framework for Side-channel Analysis
- I Choose You: Automated Hyperparameter Tuning for Deep Learning-Based Side-Channel Analysis
- The Curse of Class Imbalance and Conflicting Metrics with Machine Learning for Side-channel Evaluations