KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides, Danilo P. Mandic

Abstract

The success of Hyper-Connections (HC) in neural networks has highlighted issues related to training instability and restricted scalability. While Manifold-Constrained Hyper-Connections (mHC) mitigate these by projecting residual connections onto a Birkhoff polytope, they rely on the iterative Sinkhorn-Knopp algorithm, which is inexact and computationally expensive ($\mathcal{O}(n^3C)$). The alternative, mHC-lite, guarantees exact double stochasticity but suffers from a factorial parameter explosion ($\mathcal{O}(n!)$). We propose KromHC, which leverages Kronecker products of smaller doubly stochastic matrices. KromHC ensures exact double stochasticity while reducing parameter complexity to $\mathcal{O}(n^2C)$. Experiments show KromHC matches or outperforms state-of-the-art mHC variants with significantly fewer parameters.

Introduction: The Challenge of Scaling Residual Streams

In standard architectures like ResNets or Transformers, information flows through a single residual pathway. Hyper-Connections (HC) expand this bottleneck by splitting the residual connection into $n$ parallel streams, dynamically mixing them to increase expressivity without increasing FLOPs.

However, stacking these dynamic mixing matrices across many layers often breaks the "identity mapping" property, leading to severe training instability. Recent work introduced Manifold-Constrained HC (mHC) to enforce "double stochasticity" (rows and columns summing to 1) to preserve feature norms. Yet, mHC faces two major hurdles:

  1. Inexactness: It uses the Sinkhorn-Knopp algorithm, which only approximates double stochasticity, leading to error accumulation.
  2. Complexity: It has a high parameter complexity of $\mathcal{O}(n^3C)$.

While mHC-lite achieved exact double stochasticity using permutation matrices, its parameter count scales factorially—$\mathcal{O}(nC \cdot n!)$—making it unusable for wider residual streams.

The KromHC Method

KromHC (Kronecker-Product mHC) addresses these issues by factorizing the large $n \times n$ residual matrix into a Kronecker product of smaller matrices.

Comparison of mHC architectures. KromHC uses Kronecker products of small matrices to achieve exact double stochasticity efficiently.

The core innovation relies on the mathematical property that the Kronecker product of doubly stochastic matrices is itself doubly stochastic. By factorizing the stream width $n$ (e.g., $n=8$ into $2 \times 2 \times 2$), KromHC learns convex combinations of tiny $2 \times 2$ permutation matrices. This approach guarantees:

  • Perfect Stability: Exact double stochasticity is maintained without iterative approximations.
  • Efficiency: Parameter complexity drops to $\mathcal{O}(n^2C)$, bypassing the factorial explosion of previous methods.

Experimental Evaluation

The authors evaluated KromHC by pretraining Large Language Models (LLMs) on the FineWeb-Edu dataset at scales of 60M and 186M parameters.

Parameter Efficiency and Scalability

KromHC demonstrates superior scaling properties. For a 12-layer model with $n=4$ streams, KromHC added only 959K parameters, compared to 1.84M for standard mHC and 2.43M for mHC-lite.

Parameter complexity scaling. KromHC maintains O(n^2) growth, avoiding the factorial explosion of mHC-lite.

As shown in the figure above, while mHC-lite becomes computationally impossible as $n$ increases due to factorial growth, KromHC remains efficient even at $n=16$.

Performance Benchmarks

KromHC consistently outperformed its predecessors across multiple metrics:

  • CORE Score: Achieved a score of 16.872, surpassing mHC (16.023) and standard residual connections (14.774).
  • Reasoning Tasks: Achieved the highest average accuracy (47.7%) across commonsense reasoning benchmarks (ARC, BoolQ, PIQA, etc.).
  • Training Stability: KromHC exhibited lower and smoother gradient norms, validating the benefit of exact mathematical constraints.

Conclusion

KromHC provides a mathematically grounded solution to the stability-complexity trade-off in Hyper-Connections. By utilizing Kronecker products, it achieves the "best of both worlds": the exact training stability of permutation-based methods and the parameter efficiency required for large-scale models.

Summary updated

Abstract

The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to its training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exact doubly stochastic residual matrices; 2) mHC incurs a prohibitive $\mathcal{O}(n^3C)$ parameter complexity with $n$ as the width of the residual stream and $C$ as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, $\mathcal{O} \left( nC \cdot n! \right)$. To address both challenges, we propose \textbf{KromHC}, which uses the \underline{Kro}necker products of smaller doubly stochastic matrices to parametrize the residual matrix in \underline{mHC}. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to $\mathcal{O}(n^2C)$. Comprehensive experiments demonstrate that KromHC matches or even outperforms state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters. The code is available at \texttt{https://github.com/wz1119/KromHC}.
BibTeX
@inproceedings{zhou2026kromhc,
  author = {Wuyang Zhou and Yuxuan Gu and Giorgos Iacovides and Danilo P. Mandic},
  title = {{KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=TI7Q2o6EIa},
}
Figure