Parameter Sign Alignment in Sparse Neural Networks

18 concepts10 papers1 proposal

When the Lottery Ticket Hypothesis was first proposed by Jonathan Frankle and Michael Carbin in The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, it introduced a fascinating puzzle. They demonstrated that dense neural networks contain sparse subnetworks ("winning tickets") capable of matching the full model's performance. However, these subnetworks could only train effectively if they were reset to their exact initial weights. Re-initializing the subnetwork with new random values caused performance to collapse.

This dependency raised a fundamental question: What is the secret ingredient hidden within those precise initial floating-point values? A growing body of research reveals that the answer is surprisingly simple: it is not the exact magnitudes of the weights that matter, but their signs (+ or -).

Deconstructing the Ticket: Signs Over Magnitudes

To understand why the original lottery tickets are so sensitive to initialization, Hattie Zhou et al. deconstructed the phenomenon in Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask. They discovered that the precise numerical values of a winning ticket's initialization are largely redundant. If you reinitialize a sparse subnetwork with entirely new random magnitudes but strictly enforce sign-consistent initialization—meaning every weight retains its original positive or negative direction—the subnetwork trains just as successfully as the original winning ticket.

This insight was taken to its logical conclusion by Junghun Oh et al. in Find A Winning Sign: Sign Is All We Need to Win the Lottery. They introduced a method called A Winning Sign (AWS), proving that any arbitrary random initialization can be transformed into a winning ticket. You do not need to discover a magical set of initial weights; you simply need to overlay the correct mask and ensure the parameters inherit the correct signs. Once the directional orientation of the weights is correct, the network can easily learn the appropriate magnitudes during training.

The Optimization Intuition: Why Signs Matter

Why does the sign of a weight carry so much leverage during optimization? Think of a neural network as a complex system of directional gates. The sign of a parameter determines whether an incoming signal is amplified positively or inverted negatively.

In a fully dense network, overparameterization provides an abundance of alternative pathways and degrees of freedom. If a weight starts with the wrong sign, the surrounding network can compensate, or gradient descent can easily steer the parameter across zero to flip its sign.

However, when we prune a network down to a highly sparse architecture, we strip away these redundant pathways. In this constrained environment, flipping a weight's sign becomes exceptionally difficult because the optimization landscape develops steep barriers.

As Advait Harshal Gadhikar et al. demonstrate in Sign-In to the Lottery: Reparameterizing Sparse Training, proper sign alignment at initialization allows sparse models to bypass this struggle entirely. By starting with the correct signs, the sparse network avoids getting trapped in poor local minima and instead converges smoothly into flat, highly generalizable loss basins—mirroring the behavior of dense-to-sparse training regimes. To make training sparse networks from scratch viable, the authors introduced Sign-In, a reparameterization technique designed specifically to help sparse networks flip their signs during training without requiring an expensive dense pre-training phase.

This sign-centric perspective also explains why certain pruning algorithms outperform others. In Masks, Signs, And Learning Rate Rewinding, Advait Harshal Gadhikar and Rebekka Burkholz analyzed why Learning Rate Rewinding (LRR) consistently beats Iterative Magnitude Pruning (IMP). They found that LRR excels because it leverages the early overparameterized phase of training to establish stable, correct parameter signs. As Rebekka Burkholz notes in a companion paper, Masks, Signs, And Learning Rate Rewinding, standard pruning techniques that reset weights back to their initial random values often discard these critical, hard-won sign corrections, severely hindering the sparse network's ability to converge.

Universality Across Architectures and Paradigms

The critical role of parameter signs is not an artifact of small models or a quirk of gradient descent; it is a fundamental structural property of neural network architectures.

Challenging the Uniqueness of the Mask

If parameter signs can be aligned and adapted so effectively, how important is the actual shape of the sparse mask?

In Dual Lottery Ticket Hypothesis, Huan Wang et al. flipped the traditional lottery ticket narrative. They showed that any randomly selected sparse subnetwork can be transformed into a winning ticket using a technique called Random Sparse Network Transformation (RST). This implies that the unique connectivity pattern of a mask is less critical than the process used to condition and initialize the weights within it.

However, a note of caution is raised by Geng Yuan et al. in Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?. They observe that when winning tickets exhibit an extremely high correlation between their initial and final weights, it often indicates that the network is under-optimized and barely moving from its starting point. They argue that traditional "Pruning & Fine-tuning"—which allows weights to continue evolving rather than resetting them—consistently yields superior performance.

Together, these papers paint a cohesive picture: the classic Lottery Ticket Hypothesis works not because random initialization selects perfect numerical weights, but because it accidentally provides a subset of weights with the correct sign alignment. By focusing on preserving and optimizing these parameter signs, we can bypass expensive dense training phases and unlock the true potential of sparse neural networks.

Key Foundational Work

Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask (2019) — contributed 2 concept elements to this direction.
Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture (2020) — contributed 2 concept elements to this direction.
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_weight_sign_preservation_d89ed37a,
  title = {weight sign preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-sign-preservation/rdr_d89ed37aed27410ea8d8eb1637c263ae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_weight_sign_preservation_d89ed37a_v1,
  title = {weight sign preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-sign-preservation/rdr_d89ed37aed27410ea8d8eb1637c263ae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake