Random Initialization Variance in Deep Learning

943 concepts661 papers3 proposals

When training a deep neural network, we often treat the choice of random seed as a minor implementation detail. We initialize the weights with small random values, kick off gradient descent, and expect the data and architecture to dictate the final outcome. However, a growing body of literature reveals that this initial random state exerts a profound, often invisible influence over a model's final performance, internal logic, and generalization behavior.

To understand why, imagine the optimization landscape of a deep neural network as a vast, high-dimensional mountain range filled with thousands of local minima, valleys, and plateaus. The random seed determines where the optimization process "drops" the model into this landscape. Because the optimization path is highly non-linear, even microscopic differences in the starting coordinates can steer the model into entirely different valleys, leading to vastly different outcomes.

The Illusion of Architectural Superiority

The most immediate consequence of initialization sensitivity is the introduction of a "lottery effect" in benchmarking. When researchers report that a new architecture outperforms a baseline by a fraction of a percent, it is often impossible to tell whether the improvement stems from a genuine algorithmic breakthrough or simply a luckier random initialization.

As deep learning scaled, this variance began to collide with the margins used to claim state-of-the-art (SOTA) results. Iryna Gurevych et al. demonstrated this vulnerability in Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging, showing that the choice of random seed alone could swing an LSTM's performance by over 1%—a margin wide enough to make a mediocre configuration appear superior to a highly tuned one. This finding was echoed by Maja Popovic et al. in Exploring Variation of Results from Different Experimental Conditions, who noted that fluctuations caused by seed selection are frequently larger than the performance gains attributed to entirely new model designs.

In specialized domains, this variance can expand from a minor nuisance into a massive instability. For instance, in information retrieval, Jun Rao et al. discovered in [Where Does the Performance Improvement Come From?:

  • A Reproducibility Concern about Image-Text Retrieval](/work/where-does-the-performance-improvement-come) that standard recall metrics could swing by as much as 10 points based solely on the initialization seed. Similarly, Pedro Rodriguez et al. showed in MultiContrievers: Analysis of Dense Retrieval Representations that this initialization noise regularly eclipses the benefits of complex architectural modifications or supervised fine-tuning. When variance is this high, a single peak performance metric becomes meaningless; it represents a cherry-picked best-case scenario rather than a stable property of the architecture.

Idiosyncratic Brains: Divergent Internal Geometries

The impact of the random seed goes far deeper than the final accuracy score. Even when two identical architectures are trained on the exact same data and achieve identical accuracy, their internal "brains" can develop completely different ways of understanding the world.

By holding the data, hyperparameters, and architecture constant while varying only the random seed, Johannes Mehrer et al. isolated this phenomenon in Individual differences among deep neural network models. The authors found that different seeds cause networks to develop highly idiosyncratic internal "representational geometries" in their deeper layers.

This geometric divergence manifests as fundamentally different decision-making strategies and inductive biases:

  • Inductive Biases: Samuel Ritter et al. applied tools from cognitive psychology in Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study to show that vision models can develop entirely different preferences for object attributes (such as prioritizing shape over color) depending on their initialization.
  • Feature Attribution: This instability severely undermines model interpretability. Dylan Slack et al. demonstrated in On the Lack of Robust Interpretability of Neural Text Classifiers that two functionally identical text classifiers trained with different seeds yield significantly different feature importance rankings.
  • Representation Metrics: This lack of consistency even breaks the tools we use to measure it. Frances Ding et al. noted in Grounding Representation Similarity Through Statistical Testing that popular representation similarity metrics like CKA and CCA often fail specificity tests, outputting high distance values for models that differ only in their random initialization.

The Predictability Floor

This stochasticity creates a fundamental barrier when trying to analyze or predict model behavior before or during training. For example, scaling laws are widely used to project the performance of massive models by training much smaller prototypes. However, Leshem Choshen et al. showed in A Hitchhiker's Guide to Scaling Law Estimation that initialization variance establishes a "predictability floor." If the noise introduced by the random seed is larger than the performance differences between scaling steps, the power-law projections break down.

A similar issue plagues training-free paradigms like Zero-Shot Neural Architecture Search (NAS). Methods like the one analyzed by Guihong Li et al. in ZiCo: Zero-shot NAS via inverse Coefficient of Variation on Gradients attempt to rank candidate architectures using only their initial gradients. Because these gradients are computed at the very first step of training, they are highly sensitive to the specific random weights assigned to the network, which can lead to inconsistent and unreliable architecture rankings before training even begins.

Taming the Variance

If random initialization acts as a persistent noise floor that degrades a model's stability, how can we mitigate it? The literature points to three primary strategies: statistical rigor, parameter ensembling, and pre-training.

1. Moving Beyond Single-Point Metrics

The simplest defense against initialization variance is changing how we report results. Early work by Gerhard Paass in Assessing and Improving Neural Network Predictions by the Bootstrap Algorithm pioneered the use of resampling to generate distributions of predictions rather than single point estimates. Modern consensus dictates that researchers must report full score distributions across multiple random seeds to prove that an architectural modification provides a statistically significant benefit, rather than a lucky draw in the initialization lottery.

2. Ensembling and Fusion

Instead of trying to find a single perfect seed, we can combine multiple initializations to cancel out individual stochastic errors. Sahar E-Vahdati et al. demonstrated this in Multiple Run Ensemble Learning with Low-Dimensional Knowledge Graph Embeddings by training an ensemble of low-dimensional models in parallel, achieving better stability and accuracy than a single high-dimensional model.

For scenarios where running a full ensemble at inference time is too expensive, Ali Mammadov et al. proposed a multi-fidelity fusion strategy in Reducing Variability of Multiple Instance Learning Methods for Digital Pathology. Their approach trains several models for just a few epochs to see which seeds are performing well, and then merges their weights into a single, highly stable model, collapsing a 15 AUC-point variance down to a reliable baseline.

3. Restricting the Parameter Space via Pre-Training

The most structurally profound solution is to avoid purely random initializations altogether. In their foundational paper, The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training, Dumitru Erhan et al. demonstrated why unsupervised pre-training acts as an effective regularizer.

Purely Random Initialization:
[Random Seed] ──> Drops model anywhere in landscape ──> High Variance / Local Minima

With Pre-Training:
[Random Seed] ──> Pre-training Phase ──> Restricts parameters to a favorable region ──> Low Variance / Stable Convergence

Pre-training restricts the parameter space to a basin of attraction that contains configurations with superior generalization properties. By the time supervised fine-tuning begins, the model is no longer wandering blindly through a massive, chaotic mountain range; it has already been guided to a favorable valley where the choice of the initial random seed can no longer derail its convergence.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_initialization_sensitivity_random_seed_s_91949e13,
  title = {initialization sensitivity (random seed sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-sensitivity-random-seed-sensitivity/rdr_91949e13bd5b479291ef6ba24f701355},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_initialization_sensitivity_random_seed_s_91949e13_v1,
  title = {initialization sensitivity (random seed sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initialization-sensitivity-random-seed-sensitivity/rdr_91949e13bd5b479291ef6ba24f701355/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake