UniFormer: Unified and Efficient Transformer for Reasoning Across General and Custom Computing

Zhuoheng Ran, Chong Wu, Renjie Xu, Maolin Che, Hong Yan

TL;DR

UniFormer is a unified, hardware-friendly Transformer architecture designed to run efficiently across both general-purpose GPUs and custom reconfigurable hardware like FPGAs. By utilizing a dual-branch attention mechanism centered entirely around General Matrix Multiplication (GEMM), it achieves up to a 2.36× throughput improvement on GPUs and up to a 470× acceleration on FPGAs without sacrificing model accuracy.

What Problem Does This Paper Address?

Transformers dominate modern AI, but their heavy reliance on the self-attention mechanism introduces massive computational complexity and memory overhead. To deploy these models in low-power or real-time environments, researchers have developed numerous "efficient" Transformer variants. However, these variants are almost exclusively tailored for GPUs.

When deployed on custom hardware like FPGAs or ASICs—which rely on fundamentally different parallel computing paradigms—standard efficient Transformers stumble. They often introduce complex non-linear operations (like matrix inversions or custom activations) and irregular dataflows that are highly inefficient for custom hardware. As a result, hardware engineers are typically forced to make painful compromises between design complexity, execution speed, and model accuracy.

Methods

The authors introduce UniFormer, an architecture designed to bridge the cross-platform deployment gap using two core strategies:

  1. Dual-Branch Attention: Instead of computing a massive, full-sequence attention matrix, UniFormer splits the input sequence into two parallel branches. The global branch uses a linear-complexity attention mechanism to capture long-range token dependencies. Concurrently, the local branch uses a block-wise strategy to process fine-grained context in parallel, independent windows.
  2. GEMM as a Universal Primitive: Rather than using hybrid operators (like convolutions) that cause computational imbalance, UniFormer relies strictly on General Matrix Multiplication (GEMM). GEMM acts as the "highest common factor" for hardware optimization; it maps perfectly to both GPU Tensor Cores (via warp-level collaboration) and FPGA systolic arrays (via deeply pipelined processing elements).
Figure 3: The proposed dual-branch Transformers. The input features are split into more parallel streams and can be fused with Triton kernels with inner-loop and outer-loop scheduling strategies to maximise data and memory usage (e.g., SRAM and HBM in GPUs, BRAM and DRAM in FPGAs).

For GPU deployments, the authors developed a custom Triton-based fused kernel for the global branch to maximize on-chip SRAM data reuse and minimize slow High Bandwidth Memory (HBM) transfers. For FPGAs, the reduced block size allows the data to fit entirely within fast on-chip Block RAM (BRAM), eliminating off-chip memory bottlenecks.

Datasets

The model's accuracy and throughput were evaluated on the standard ImageNet classification dataset (224×224 resolution) to provide a fair comparison against other state-of-the-art Vision Transformers (ViTs).

Key Insights and Ideas

  • The Power of the "Highest Common Factor": While many papers try to replace attention with hardware-unfriendly approximations, sticking to GEMM preserves mathematical fidelity. Because GEMM is already heavily optimized at the silicon level across diverse architectures, it serves as the ideal primitive for cross-platform portability.
  • More Customization Isn't Always Better: Counterintuitively, writing custom hardware kernels for every part of a network can hurt performance. The authors discovered that combining PyTorch's native, highly optimized local attention path with their custom Triton global kernel (Local-PyTorch+Global-Kernel) actually outperformed a fully customized kernel implementation (Uni_Flash_Mix). Over-fusing compute and storage can introduce downsides like kernel contention and scheduling overhead.
  • Breaking the Quadratic Barrier on FPGAs: Vanilla attention scales quadratically ($O(N^2)$) with input size, causing FPGA latency to skyrocket as sequences grow. UniFormer flattens this curve into a nearly linear growth pattern, making Transformers viable for edge hardware processing longer contexts.

Key Results

  • GPU Performance: Evaluated on an NVIDIA H20 GPU against competitive baselines (like EfficientViT and Flatten Transformer), UniFormer achieved a throughput of 4280 FPS—a 1.79× to 2.36× speedup—while maintaining a top-tier 82.9% Top-1 accuracy on ImageNet.
  • FPGA Performance: When implemented on a Xilinx Zynq UltraScale+ FPGA using Vitis HLS, UniFormer delivered a massive 180× to 470× acceleration and corresponding energy efficiency gains over a vanilla Transformer baseline.

Limitations and Future Work

While the introduction frames UniFormer as a solution for generative AI and Large Language Models (LLMs), the empirical validation in the paper is strictly limited to vision tasks (ImageNet classification) using relatively compact models (20M–21M parameters). The paper does not discuss how this dual-branch GEMM architecture fares at true LLM scale (billions of parameters) or how it handles autoregressive decoding workloads, leaving that as an open direction for future research.

Summary updated

Abstract

The success of neural networks such as convolutional neural networks (CNNs) has been largely attributed to their effective and widespread deployment on customised computing platforms, including field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs). In the current era, Transformer-based architectures underpin the majority of state-of-the-art (SOTA) larger models that are also increasingly deployed on customised computing hardware for low-power and real-time applications. However, the fundamentally different parallel computation paradigms between general-purpose and customised computing often lead to compromises in model transfer and deployability, which typically come at the cost of complexity, efficiency or accuracy. Moreover, many cross-platform optimisation principles have also remained underexplored in existing studies. This paper introduces UniFormer, a unified and efficient Transformer architecture for both general-purpose and customised computing platforms. By enabling higher parallelism and compute-storage fusion, UniFormer achieves state-of-the-art (SOTA) accuracy and latency on GPUs while exhibiting strong adaptability on FPGAs. To the best of our knowledge, this paper is the first efficient Transformer work that jointly considers both general-purpose and customised computing architectures.
BibTeX
@misc{ran2025uniformer,
  author = {Zhuoheng Ran and Chong Wu and Renjie Xu and Maolin Che and Hong Yan},
  title = {{UniFormer: Unified and Efficient Transformer for Reasoning Across General and Custom Computing}},
  year = {2025},
  eprint = {2511.08135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.08135},
}
Figure