Initializing Constrained Neural Networks for Stable Signal Propagation

This direction is merged into another direction: monotonicity-constrained neural networks. Its last version stays readable and citable here.

Version 1 of 18 papers20 conceptsfrozen 2026-10-04

In standard deep learning, we typically allow weights to take any real value. However, many high-stakes applications require neural networks to obey strict mathematical laws. For example, in combinatorial auctions, a bidder’s value for a set of items should never decrease as more items are added—a property known as monotonicity. Similarly, in physics-informed machine learning or optimization, we often require Input-Convex Neural Networks (ICNNs) to guarantee global convergence.

To enforce these properties, researchers often use specialized architectures where weights are constrained to be non-negative. While mathematically elegant, these constraints break the fundamental tool we use to train deep networks: standard weight initialization.

The Zero-Mean Assumption

The success of modern deep learning relies on "signal propagation"—the idea that the variance of activations should remain roughly constant as data flows through the layers. If the variance grows, activations explode; if it shrinks, they vanish.

Standard methods like Xavier or Kaiming initialization achieve this by assuming weights are sampled from a distribution centered at zero. When weights are zero-mean, the positive and negative signals cancel each other out just enough to keep the total variance stable.

As Pieter-Jan Hoedt and Günter Klambauer point out in Principled Weight Initialisation for Input-Convex Neural Networks, this assumption is fundamentally incompatible with constrained networks. If you force weights to be non-negative to ensure convexity or monotonicity, you can no longer have a zero-mean distribution. Without the "canceling out" effect of negative weights, the signal accumulates layer by layer, leading to an exponential explosion of activations.

The Problem of Exploding Activations

In the context of combinatorial auctions, Jakob Heiss, Jakob Weissteiner, and their colleagues encountered this exact wall while developing the BOCA framework. In Bayesian Optimization-based Combinatorial Assignment, they observed that standard initialization techniques caused monotone neural networks to suffer from vanishing or exploding gradients, making them nearly impossible to train for complex value estimation.

When weights are restricted to non-negative values, the "mean" of the weight distribution shifts positively. In a deep network, this positive bias acts like a compound interest rate on the input signal. By the time the signal reaches the final layers, it has grown so large that the gradients become numerically unstable. This is not just a minor inefficiency; it is a structural failure of the training process for architectures like ICNNs and monotone networks.

Stabilizing Signal Propagation with Non-Zero Means

To solve this, researchers have moved toward a new theory of signal propagation that explicitly accounts for non-zero mean weight distributions. The goal is to find a "fixed point"—a specific set of distribution parameters where the mean and variance of activations remain stable across layers despite the non-negativity constraint.

  1. Mixture Distributions: In their work on Bayesian Optimization-Based Combinatorial Assignment, Julien Siems and Sven Seuken utilize a mixture-distribution approach. By carefully balancing the weights, they prevent the exponential growth of neuron activations, allowing deep monotone networks to converge on optimal auction allocations.
  2. Log-Normal Sampling: Pieter-Jan Hoedt and Günter Klambauer propose sampling weights from a log-normal distribution. By deriving the exact mean and variance needed to satisfy the non-negativity constraint while maintaining signal variance, they enable ICNNs to train faster and generalize better, even without the help of skip-connections.
  3. Robust Normalization: In scientific domains like high-energy physics, Ouail Kitouni, Niklas Nolte, and Mike Williams argue in Robust and Provably Monotonic Networks that standard Xavier/Kaiming schemes are unoptimized for networks using weight normalization. They emphasize that specialized initialization is the bridge that allows high-performance deep learning to meet the rigorous, physically sensible constraints required for the Large Hadron Collider.

Why It Works

The intuition behind these new initialization strategies is a shift from symmetry to balance. Standard initialization relies on the symmetry of the weight distribution around zero to keep the signal in check. Principled initialization for constrained networks relies on a mathematical balance: it calculates exactly how much the non-negative weights will "push" the signal and adjusts the initial variance to "pull" it back, ensuring the signal neither disappears into nothingness nor explodes into infinity.

By moving beyond zero-centered assumptions, these researchers have unlocked the ability to train deep, provably constrained models that are as stable and powerful as their unconstrained counterparts.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_initializing_constrained_neural_networks_f2d1fa24,
  title = {Initializing Constrained Neural Networks for Stable Signal Propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initializing-constrained-neural-networks-for-stable-signal-propagation/rdr_f2d1fa24a01443e4b23275dc3149c474},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_initializing_constrained_neural_networks_f2d1fa24_v1,
  title = {Initializing Constrained Neural Networks for Stable Signal Propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/initializing-constrained-neural-networks-for-stable-signal-propagation/rdr_f2d1fa24a01443e4b23275dc3149c474/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}