Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

Mohammad Mahdi Moradi, Walid Ahmed, S. X. Wen, Sudhir P. Mudur, Weiwei Zhang, Yang LiuarXiv 2025 · DOI · arXiv

Abstract

FlowHN is a novel parallel hybrid neural network that combines the fine-grained reasoning of Transformers with the long-range efficiency of State-Space Models (SSMs). By using a "FLOP-aware circulating split" to dynamically balance the workload between parallel branches, it achieves up to 4× higher throughput and 2× better hardware utilization than existing hybrid models while maintaining superior accuracy. This work addresses the critical trade-offs in hybrid architectures: synchronization bottlenecks and representation divergence.

Introduction

Modern Large Language Models (LLMs) face a fundamental trade-off between expressivity and efficiency. Standard Attention mechanisms are excellent at capturing complex, fine-grained token interactions but suffer from quadratic complexity ($O(N^2)$), making them slow and memory-hungry for long sequences. In contrast, State-Space Models (SSMs), like Mamba, offer linear scaling and high efficiency but often struggle with the precise information retrieval that Transformers handle with ease.

Previous attempts to hybridize these two have encountered significant hurdles. Sequential hybrids stack Attention and SSM layers, creating bottlenecks where hardware for one component sits idle while the other processes. Parallel hybrids run both side-by-side but face load imbalance—where the heavier Attention branch forces the faster SSM branch to wait—and representation divergence, making it difficult to merge their distinct information types without knowledge loss.

Methodology

The authors introduce FlowHN, a parallel hybrid architecture designed to solve these synchronization and fusion issues. The core innovation lies in how tokens are distributed and merged within a parallel block.

FlowHN Parallel Architecture

As shown in the architecture diagram, FlowHN utilizes a parallel structure with two primary innovations:

  • FLOP-Aware Circulating Split (FAC_Split): Instead of sending all tokens to both branches, FlowHN partitions the input sequence based on computational cost. It calculates the FLOPs of the Attention vs. the SSM branch and distributes tokens inversely proportional to those costs. This ensures both branches finish simultaneously, maximizing hardware efficiency.
  • Circulating Logic: To prevent information loss, token assignments rotate across different blocks. Over the course of the network, every token eventually passes through both Attention and SSM branches, ensuring comprehensive feature extraction.
  • Token Fusion: After parallel processing, the divergent outputs are concatenated and passed through a linear projection. This "semantically enriched" embedding integrates global context from Attention with the recurrent memory of the SSM.

Models were trained on SlimPajama-6B, a diverse 6-billion-token English corpus, and evaluated across standard benchmarks including ARC, PIQA, HellaSwag, BoolQ, and Winogrande.

Results and Discussion

FlowHN was tested at 135M, 350M, and 1B parameter scales against state-of-the-art models like LLaMA, Mamba-2, Jamba, and Hymba.

Performance Comparison across Model Scales

Key findings include:

  • Throughput (TPS): FlowHN achieved up to 4× higher Tokens Per Second compared to sequential hybrid models. At the 1B scale, the FAC_Split variant sustained over 7,900 TPS, significantly outperforming baselines.
  • Efficiency (MFU): It delivered 2× better hardware utilization. For the 1B model, FAC_Split reached a Model FLOPs Utilization (MFU) of 47.63%, compared to Hymba’s 33.16%.
  • Accuracy: Despite the token splitting, FlowHN consistently outperformed or matched competitors. At the 350M scale, the No_split version achieved a 41.31% average accuracy, leading its closest parallel competitor, Hymba, by 1.5%.

These results underscore that hardware utilization is as critical as theoretical complexity. By synchronizing completion times through proportional loading, FlowHN minimizes idle periods and maximizes GPU efficiency.

Conclusion

FlowHN demonstrates that parallel hybrid architectures can overcome the limitations of sequential designs through intelligent load balancing and fusion. While currently validated on autoregressive modeling and commonsense reasoning, future work may explore dynamic routing based on token difficulty and scaling the architecture to the 7B+ parameter range to further test its efficiency gains.

Summary updated

Abstract

Attention and State-Space Models (SSMs) when combined in a hybrid network in sequence or in parallel provide complementary strengths. In a hybrid sequential pipeline they alternate between applying a transformer to the input and then feeding its output into a SSM. This results in idle periods in the individual components increasing end-to-end latency and lowering throughput caps. In the parallel hybrid architecture, the transformer operates independently in parallel with the SSM, and these pairs are cascaded, with output from one pair forming the input to the next. Two issues are (i) creating an expressive knowledge representation with the inherently divergent outputs from these separate branches, and (ii) load balancing the computation between these parallel branches, while maintaining representation fidelity. In this work we present FlowHN, a novel parallel hybrid network architecture that accommodates various strategies for load balancing, achieved through appropriate distribution of input tokens between the two branches. Two innovative differentiating factors in FlowHN include a FLOP aware dynamic token split between the attention and SSM branches yielding efficient balance in compute load, and secondly, a method to fuse the highly divergent outputs from individual branches for enhancing representation expressivity. Together they enable much better token processing speeds, avoid bottlenecks, and at the same time yield significantly improved accuracy as compared to other competing works. We conduct comprehensive experiments on autoregressive language modeling for models with 135M, 350M, and 1B parameters. FlowHN outperforms sequential hybrid models and its parallel counterpart, achieving up to 4* higher Tokens per Second (TPS) and 2* better Model FLOPs Utilization (MFU).
BibTeX
@misc{moradi2025balancing,
  author = {Mohammad Mahdi Moradi and Walid Ahmed and S. X. Wen and Sudhir P. Mudur and Weiwei Zhang and Yang Liu},
  title = {{Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks}},
  year = {2025},
  eprint = {2505.19472},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19472},
  howpublished = {arXiv},
}
Figure