Abstract
The success of Hyper-Connections (HC) in neural networks has highlighted issues related to training instability and restricted scalability. While Manifold-Constrained Hyper-Connections (mHC) mitigate these by projecting residual connections onto a Birkhoff polytope, they rely on the iterative Sinkhorn-Knopp algorithm, which is inexact and computationally expensive ($\mathcal{O}(n^3C)$). The alternative, mHC-lite, guarantees exact double stochasticity but suffers from a factorial parameter explosion ($\mathcal{O}(n!)$). We propose KromHC, which leverages Kronecker products of smaller doubly stochastic matrices. KromHC ensures exact double stochasticity while reducing parameter complexity to $\mathcal{O}(n^2C)$. Experiments show KromHC matches or outperforms state-of-the-art mHC variants with significantly fewer parameters.
Introduction: The Challenge of Scaling Residual Streams
In standard architectures like ResNets or Transformers, information flows through a single residual pathway. Hyper-Connections (HC) expand this bottleneck by splitting the residual connection into $n$ parallel streams, dynamically mixing them to increase expressivity without increasing FLOPs.
However, stacking these dynamic mixing matrices across many layers often breaks the "identity mapping" property, leading to severe training instability. Recent work introduced Manifold-Constrained HC (mHC) to enforce "double stochasticity" (rows and columns summing to 1) to preserve feature norms. Yet, mHC faces two major hurdles:
- Inexactness: It uses the Sinkhorn-Knopp algorithm, which only approximates double stochasticity, leading to error accumulation.
- Complexity: It has a high parameter complexity of $\mathcal{O}(n^3C)$.
While mHC-lite achieved exact double stochasticity using permutation matrices, its parameter count scales factorially—$\mathcal{O}(nC \cdot n!)$—making it unusable for wider residual streams.
The KromHC Method
KromHC (Kronecker-Product mHC) addresses these issues by factorizing the large $n \times n$ residual matrix into a Kronecker product of smaller matrices.
The core innovation relies on the mathematical property that the Kronecker product of doubly stochastic matrices is itself doubly stochastic. By factorizing the stream width $n$ (e.g., $n=8$ into $2 \times 2 \times 2$), KromHC learns convex combinations of tiny $2 \times 2$ permutation matrices. This approach guarantees:
- Perfect Stability: Exact double stochasticity is maintained without iterative approximations.
- Efficiency: Parameter complexity drops to $\mathcal{O}(n^2C)$, bypassing the factorial explosion of previous methods.
Experimental Evaluation
The authors evaluated KromHC by pretraining Large Language Models (LLMs) on the FineWeb-Edu dataset at scales of 60M and 186M parameters.
Parameter Efficiency and Scalability
KromHC demonstrates superior scaling properties. For a 12-layer model with $n=4$ streams, KromHC added only 959K parameters, compared to 1.84M for standard mHC and 2.43M for mHC-lite.
As shown in the figure above, while mHC-lite becomes computationally impossible as $n$ increases due to factorial growth, KromHC remains efficient even at $n=16$.
Performance Benchmarks
KromHC consistently outperformed its predecessors across multiple metrics:
- CORE Score: Achieved a score of 16.872, surpassing mHC (16.023) and standard residual connections (14.774).
- Reasoning Tasks: Achieved the highest average accuracy (47.7%) across commonsense reasoning benchmarks (ARC, BoolQ, PIQA, etc.).
- Training Stability: KromHC exhibited lower and smoother gradient norms, validating the benefit of exact mathematical constraints.
Conclusion
KromHC provides a mathematically grounded solution to the stability-complexity trade-off in Hyper-Connections. By utilizing Kronecker products, it achieves the "best of both worlds": the exact training stability of permutation-based methods and the parameter efficiency required for large-scale models.
Summary updated