Cheap Model Selection That Survives Market Crashes

Forecasting how volatile a market will be tomorrow is an old problem with a lopsided structure. A 5% drop raises future volatility much more than a 5% rise, an asymmetry economists call the leverage effect. Recent hybrid models bolt a small neural network onto a classical volatility equation and beat the classical models mainly during crises. To choose such a network automatically, one would borrow cost-cutting tools from neural architecture search (the automated search over network designs): train each candidate briefly, on less data, or score it from an early checkpoint, and keep the ones that rank well.

This note argues that these shortcuts fail in a specific way on volatility data. A cheap evaluation that never contains a large negative shock cannot tell apart architectures whose only difference is how they react to bad news. Those are exactly the architectures that matter for risk management. We propose to measure this failure on a small, fully trained benchmark of volatility architectures. We then test a simple fix: choose the cheap evaluation budget, and the data it sees, so that it preserves the ranking of models during stress windows rather than on average.

Core Claim

Cheap performance estimators are usually tuned to maximize the overall rank correlation between cheap and full evaluation. On volatility data, that overall correlation is dominated by calm periods, which make up most trading days. We hypothesize three things:

  1. Low-fidelity rankings will correlate well with full-training rankings when both are measured on calm-period out-of-sample likelihood.
  2. The same rankings will correlate much worse on stress-period likelihood.
  3. The misranking will concentrate on candidates that carry explicit asymmetric components, such as inputs that split returns into positive and negative parts, or a threshold term that lets negative shocks enter the variance equation separately.

The fix tests whether a stress-aware choice of budget recovers the right ranking at the same cost. In that fix, the cheap evaluation always includes historical shock episodes, and the budget is selected by rank correlation on stress windows. "Stress" is defined by realized volatility jumps rather than by the sign of returns, so the method also applies where the asymmetry runs in reverse.

Grounding in Prior Work

Several papers motivate the gap.

Other papers supply the method or constrain it.

Research Program

The search space

We will build a compact tabular benchmark around the LSTM-BEKK structure. The static and persistence terms are kept, and the search runs over the neural dynamic term along four axes:

  • Backbone family: LSTM, GRU, dilated temporal convolution, an MLP on lagged returns, or a one-layer causal attention block.
  • Size: depth and hidden width.
  • Input encoding: raw returns, or returns split into positive and negative parts.
  • Threshold term: whether the persistence component includes a GJR-style term, where squared shocks enter separately when the shock is negative.

The last two axes are the "asymmetry operators" whose ranking we expect cheap estimators to garble. The space is kept to roughly a thousand configurations so that every one can be trained to convergence with three seeds. Volatility models of this size are small, and we will report the actual compute rather than assume it.

Data and evaluation windows

The data are daily returns on the U.S., U.K., and Japan 100-asset universes over 2014 to 2023, matching the setup of Deep Learning Enhanced Multivariate GARCH. We add CSI 300 constituents as a reverse-asymmetry stress test, since Study on the Volatility of CSI Index Returns 300 Based on GARCH Modeling covers that index.

Evaluation uses rolling origins. Windows are labeled "stress" when realized volatility rises above a trailing quantile, and "calm" otherwise. Stress windows include the 2015 to 2016 selloff, late 2018, and March 2020. A later drawdown is held out entirely for final testing.

Cheap evaluations

The cheap evaluations vary three things: the number of epochs, the length of the training history, and the composition of the validation set.

The stress-aware variant has two parts. First, the data it sees always retains the labeled shock episodes, even when the training history is truncated. Second, its budget is chosen by MIP to maximize Spearman correlation with full-training negative log-likelihood (NLL) on stress windows only.

Baselines

Full training of every candidate. This is the naive reference. It is also the ground truth, and its cost is the denominator for every speedup we report.

Classical models with no search. This is the econometric reference: Scalar BEKK, DCC, and a GJR-style asymmetric BEKK, each fitted by maximum likelihood on the same windows. Any searched model has to beat these on stress-window NLL to justify search at all. The cited paper reports that LSTM-BEKK cut maximum drawdown on the U.S. 250-asset minimum-variance portfolio to -0.161 against DCC's -0.260, and that LSTM-BEKK was the only model retained in the 90% model confidence set across nine settings (Deep Learning Enhanced Multivariate GARCH). No exact NLL values for that setting are reported on Lacuna.

Hand-designed LSTM-BEKK. This is the strongest existing method and is reimplemented at its published architecture. Our selected model counts as beating it if it lowers stress-window NLL and reduces drawdown on the same universes and splits.

Standard BPE/MIP. This is the strongest cheap-selection competitor. It is the same random-forest pruning procedure from Rethinking Performance Estimation in Neural Architecture Search, targeting overall rank correlation, and is given an identical compute budget and the same sample of architectures to calibrate on. On CIFAR-10 that paper reports search with BPE-1 at 0.33 GPU days against 3,150 for AmoebaNet. No reported result exists on Lacuna for BPE on volatility data.

Early validation loss as a covariate. This is the one-third-mark covariate from Can Training Logs Make Model Comparisons More Precise?, run at both 10 and 50 seeds per arm.

Ablation of the key ingredient. The two parts of the stress-aware variant are switched on separately. One arm only retains shock episodes in the data and still selects the budget on overall correlation. The other arm only selects the budget on stress correlation and does not force shock episodes into the data. This separates data composition from budget selection.

Beating standard BPE means three things at equal cost: higher Kendall correlation on stress windows, better recovery of the true top 10%, and lower stress-window regret for the architecture finally chosen.

Why not just…?

Why not just fit an asymmetric GARCH and skip neural search? These models are instant to fit and build in the leverage effect. Against them, LSTM-BEKK beat Scalar BEKK and DCC out of sample, with the gap concentrated in the 2020 crash (Deep Learning Enhanced Multivariate GARCH). The GJR-style BEKK baseline settles the question directly. If it matches the searched models on stress-window NLL, we will report that search is not worth doing here.

Why not just train every candidate fully, since these models are small? This is a real threat. If full training of the whole space is cheap at 100 assets, cheap estimators matter little there. We will report the wall-clock cost of exhaustive training and how it scales with asset count. The benchmark built by exhaustive training is useful regardless, but the claim about cheap selection only matters in the regime where exhaustive training stops being affordable.

Why not just put crisis periods into the validation set and use ordinary early stopping? This may capture most of the benefit. The data-composition-only arm of the ablation is exactly this alternative. If it matches the full stress-aware variant, the contribution reduces to that simpler recipe, and we will say so.

Why not just use zero-cost proxies or a weight-sharing supernet? Both fail to separate the top 10% even on images (EZNAS). Supernet rankings fall to near zero on a recurrent space (How Does Supernet Help in Neural Architecture Search?). We will include NASWOT and Synflow scores as cheap reference points, but we do not expect them to resolve differences in how models respond to shocks.

Decisive Checks

The first check is the diagnosis itself. On the tabular benchmark, standard BPE should show a clear gap between its calm-window and stress-window rank correlations. The architectures it misranks should be enriched for sign-split inputs and threshold terms. If no gap appears, the premise fails and the fix is unnecessary.

The second check asks whether the stress-aware variant closes that gap at equal cost. It should also pick an architecture whose held-out drawdown NLL and minimum-variance portfolio drawdown are no worse than full-training selection.

The third check uses CSI 300 constituents. Because stress is defined by volatility jumps rather than sign, the method should select models whose learned asymmetry follows the local reverse pattern rather than the textbook one.