TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

Xinying Hu, You Wu, Haoyi Wu, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Kewei TuarXiv 2026 · arXiv

TL;DR

TPHC (Tensor-Parallel grouped Hyper-Connections) supercharges Transformer models by expanding the single residual stream into multiple parallel paths that actually stay diverse. By repurposing the partial outputs already computed during tensor-parallel training, TPHC achieves better reasoning performance and lower loss than previous methods with minimal computational overhead.

What Problem Does This Paper Address?

Standard Transformers rely on a single residual stream to pass information across layers. Recent research into Hyper-Connections (HC) proposed expanding this to $N$ parallel streams to improve information flow. However, these models often suffer from "stream collapse": because every stream receives a copy of the exact same sublayer output, the parallel paths quickly become highly correlated and redundant.

Previous attempts to fix this, like xHC, added extra modules (like causal convolutions) to create diversity, but this increases complexity. The authors of TPHC asked a clever question: Can we get this diversity for "free" from the way we already train large models?

Methods

The core innovation of TPHC is integrating Tensor Parallelism (TP)—a standard technique for splitting large models across multiple GPUs—directly into the Hyper-Connection architecture.

In a standard TP setup, a sublayer is split into $k$ shards. Each shard computes a partial output, and these are summed together (via an "all-reduce" operation) before being added back to the residual stream. TPHC changes this flow:

  1. Grouped Reads: Instead of every shard reading the same input, each shard (or group) reads its own learned mixture of the $N$ residual streams.
  2. Grouped Writes: Instead of summing the shard outputs into one, TPHC keeps them separate. Each shard's output is written back to a different mixture of the residual streams.
  3. Sparse Updates: To keep costs down, each sublayer only writes to a subset of the streams ($k$ out of $N$) using a fixed "sliding" schedule.
  4. Identity Write Initialization: The authors initialize the model so that each group primarily interacts with its own dedicated stream, breaking the symmetry that usually leads to stream collapse.
Comparison between standard manifold-constrained HC (mHC) and the proposed TPHC. While mHC collapses sublayer outputs into one, TPHC keeps them separate to maintain stream diversity.

To make this efficient at scale, the authors also implement sharded streams, where the residual streams themselves are partitioned across GPUs, reducing memory pressure and communication bottlenecks.

Figure 2: One sublayer of (a) mHC and (b) TPHC un- der tensor parallelism of degree 4 with sharded streams. GPU j holds stream j and shard j of the sublayer, bars are collectives, and orange marks where (a) and (b) dif- fer. mHC all-reduces the read, since every shard needs the full input u, and its write sums the partial outputs oj. TPHC reduce-scatters the read into the group inputs uj and writes the oj as separate components.

Datasets

The researchers evaluated TPHC using:

  • Training: A 1.1B parameter model trained on 100B tokens of the SlimPajama dataset.
  • Ablations: Smaller runs on the MiniPile dataset (1.7B tokens).
  • Benchmarks: Six zero-shot commonsense reasoning tasks, including HellaSwag, OpenBookQA, WinoGrande, PIQA, and the ARC challenges.

Key Insights and Ideas

  • Diversity is Efficiency: The "aha moment" is realizing that the partial outputs in tensor parallelism are a wasted source of information diversity. By not summing them immediately, the model can learn more complex features across its parallel streams.
  • Breaking Symmetry Matters: Standard initialization makes all streams identical. TPHC’s "identity write" initialization forces the streams to start different and stay different, which is crucial for performance.
  • Scaling Residuals: Unlike previous HC methods that hit a wall at 4 streams, TPHC continues to improve as you add more streams (up to 16 tested), largely because it prevents the streams from converging to a single direction.
Analysis of stream diversity. TPHC4 (blue) maintains significantly lower cosine similarity between streams compared to mHC4 (grey), proving it successfully avoids stream collapse.

Key Results

  • Lower Loss: On a 1.1B model, TPHC reduced validation loss by 0.7–1.0% compared to the previous state-of-the-art manifold-constrained HC (mHC).
  • Better Reasoning: TPHC improved average zero-shot accuracy by 0.3–0.7 percentage points across reasoning benchmarks.
  • Superior Scaling: Remarkably, a TPHC model with only 8 streams outperformed an mHC model with 16 streams.
  • Low Overhead: Despite the architectural changes, TPHC is highly efficient. On an 8B model, it only adds 9% to training time compared to mHC when using 4 streams.
Figure 3: Training loss of the 1.1B hyper-connection models minus that of Llama, smoothed over 1k steps. Color marks the method and line style the number of streams N. Lower is better.

Limitations and Future Work

  • Architecture Specificity: TPHC is designed around Tensor Parallelism in dense Transformers. The authors note that it isn't immediately clear how to transfer this design to Mixture-of-Experts (MoE) models that use Expert Parallelism.
  • Training Scale: While 100B tokens is significant, it is small compared to the trillions of tokens used for frontier models. The long-term stability and benefits at "Llama-3 scale" remain to be fully explored.
  • Future Directions: The authors suggest extending TPHC to MoE models as a promising next step.

Summary updated

Abstract

Hyper-Connections (HC) expand the residual stream of a Transformer into $N$ parallel streams, and manifold-constrained HC (mHC) stabilizes them at scale. However, mHC benefits little from more than four streams, partly because each sublayer writes a single output to all streams, limiting how differently they can be updated. In this work, we propose TPHC, which integrates tensor-parallel (TP) shards with HC streams to promote greater independence across streams. Specifically, we route a separate mixture of the residual streams to each TP shard and keep the individual shard outputs separate during write-back, so each stream can receive a different mixture of those outputs. This design makes the residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers. With native support for stream sharding under TP, TPHC requires only 9% more training time than mHC on an 8B model when both use four streams. Increasing the number of streams from four to sixteen raises its training time by only 19%. On a 1.1B model trained on 100B tokens, TPHC reduces validation loss by 0.7-1.0% and raises average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3-0.7 percentage points relative to mHC at matched stream counts $N\in\{4,8,16\}$. Our code is available at https://github.com/wuyou2002/TPHC.
BibTeX
@misc{hu2026tphc,
  author = {Xinying Hu and You Wu and Haoyi Wu and Jianchao Tan and Kefeng Zhang and Yuchen Xie and Kewei Tu},
  title = {{TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates}},
  year = {2026},
  eprint = {2611.23026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23026},
}
Figure