Abstract
FlowHN is a novel parallel hybrid neural network that combines the fine-grained reasoning of Transformers with the long-range efficiency of State-Space Models (SSMs). By using a "FLOP-aware circulating split" to dynamically balance the workload between parallel branches, it achieves up to 4× higher throughput and 2× better hardware utilization than existing hybrid models while maintaining superior accuracy. This work addresses the critical trade-offs in hybrid architectures: synchronization bottlenecks and representation divergence.
Introduction
Modern Large Language Models (LLMs) face a fundamental trade-off between expressivity and efficiency. Standard Attention mechanisms are excellent at capturing complex, fine-grained token interactions but suffer from quadratic complexity ($O(N^2)$), making them slow and memory-hungry for long sequences. In contrast, State-Space Models (SSMs), like Mamba, offer linear scaling and high efficiency but often struggle with the precise information retrieval that Transformers handle with ease.
Previous attempts to hybridize these two have encountered significant hurdles. Sequential hybrids stack Attention and SSM layers, creating bottlenecks where hardware for one component sits idle while the other processes. Parallel hybrids run both side-by-side but face load imbalance—where the heavier Attention branch forces the faster SSM branch to wait—and representation divergence, making it difficult to merge their distinct information types without knowledge loss.
Methodology
The authors introduce FlowHN, a parallel hybrid architecture designed to solve these synchronization and fusion issues. The core innovation lies in how tokens are distributed and merged within a parallel block.
As shown in the architecture diagram, FlowHN utilizes a parallel structure with two primary innovations:
- FLOP-Aware Circulating Split (FAC_Split): Instead of sending all tokens to both branches, FlowHN partitions the input sequence based on computational cost. It calculates the FLOPs of the Attention vs. the SSM branch and distributes tokens inversely proportional to those costs. This ensures both branches finish simultaneously, maximizing hardware efficiency.
- Circulating Logic: To prevent information loss, token assignments rotate across different blocks. Over the course of the network, every token eventually passes through both Attention and SSM branches, ensuring comprehensive feature extraction.
- Token Fusion: After parallel processing, the divergent outputs are concatenated and passed through a linear projection. This "semantically enriched" embedding integrates global context from Attention with the recurrent memory of the SSM.
Models were trained on SlimPajama-6B, a diverse 6-billion-token English corpus, and evaluated across standard benchmarks including ARC, PIQA, HellaSwag, BoolQ, and Winogrande.
Results and Discussion
FlowHN was tested at 135M, 350M, and 1B parameter scales against state-of-the-art models like LLaMA, Mamba-2, Jamba, and Hymba.
Key findings include:
- Throughput (TPS): FlowHN achieved up to 4× higher Tokens Per Second compared to sequential hybrid models. At the 1B scale, the
FAC_Splitvariant sustained over 7,900 TPS, significantly outperforming baselines. - Efficiency (MFU): It delivered 2× better hardware utilization. For the 1B model,
FAC_Splitreached a Model FLOPs Utilization (MFU) of 47.63%, compared to Hymba’s 33.16%. - Accuracy: Despite the token splitting, FlowHN consistently outperformed or matched competitors. At the 350M scale, the
No_splitversion achieved a 41.31% average accuracy, leading its closest parallel competitor, Hymba, by 1.5%.
These results underscore that hardware utilization is as critical as theoretical complexity. By synchronizing completion times through proportional loading, FlowHN minimizes idle periods and maximizes GPU efficiency.
Conclusion
FlowHN demonstrates that parallel hybrid architectures can overcome the limitations of sequential designs through intelligent load balancing and fusion. While currently validated on autoregressive modeling and commonsense reasoning, future work may explore dynamic routing based on token difficulty and scaling the architecture to the 7B+ parameter range to further test its efficiency gains.
Summary updated