Gaussian Blending Unit: An Edge GPU Plug-in for Real-Time Gaussian-Based Rendering in AR/VR

Zhifan Ye, Yonggan Fu, Jingqun Zhang, Leshu Li, Yongan Zhang, Sixu Li, Cheng Wan, Chenxi Wan, Chaojian Li, Sreemanth Prathipati, Yingyan Celine Lin

Abstract

3D Gaussian Splatting (3D-GS) has emerged as a premier method for photorealistic 3D scene representation, yet its computational demands prevent real-time performance on resource-constrained edge devices. This paper presents the Gaussian Blending Unit (GBU), a specialized hardware plug-in designed to accelerate the Gaussian blending stage—the primary bottleneck in the 3D-GS pipeline. By introducing a novel Intra-Row Sequential Shading (IRSS) dataflow and a row-centric hardware architecture, the GBU reduces redundant computations and improves hardware utilization. Experimental results on edge GPUs demonstrate that the GBU enables rendering speeds exceeding 60 FPS across static, dynamic, and avatar scenes while significantly improving energy efficiency.

Introduction

Immersive Augmented and Virtual Reality (AR/VR) applications demand high-fidelity rendering at frame rates of at least 60 FPS to ensure user comfort and presence. While Neural Radiance Fields (NeRFs) initially set the standard for quality, 3D Gaussian Splatting (3D-GS) has recently surpassed them by offering a superior balance of visual fidelity and rendering speed. However, this performance is largely confined to high-end desktop GPUs. On edge GPUs, such as the NVIDIA Jetson Orin NX typically found in mobile headsets, 3D-GS performance drops to an unplayable 7–17 FPS.

Profiling the 3D-GS pipeline reveals that the Gaussian Blending Stage is the critical bottleneck, accounting for 48% to 78% of total rendering time. This stage is computationally expensive and poorly suited for the Single Instruction, Multiple Threads (SIMT) architecture of standard GPUs. This paper introduces the GBU to address these inefficiencies through a hardware-software co-design approach.

The Blending Bottleneck

The inefficiency of Gaussian blending on standard GPUs stems from two primary factors:

  1. Computational Complexity: Calculating the opacity and color contribution of every Gaussian to every pixel requires approximately 11 floating-point operations (FLOPs) per fragment.
  2. SIMT Inefficiency: GPUs process pixels in lockstep warps. Because Gaussians overlap irregularly across a scene, different pixel rows within a tile often have vastly different workloads. This imbalance leads to low compute utilization, measured at approximately 18.9% on edge platforms.

The GBU Architecture

Intra-Row Sequential Shading (IRSS)

To minimize redundant math, the authors developed the IRSS dataflow. This method processes pixel rows sequentially rather than in parallel tiles. By applying a coordinate transformation—utilizing eigenvalue decomposition and rotation—anisotropic 2D Gaussians are transformed into isotropic circles aligned with the x-axis. This mathematical optimization allows the system to share intermediate values between adjacent pixels, reducing the computational load from 11 FLOPs to just 2 FLOPs per fragment. Furthermore, because Gaussians are convex, IRSS can precisely identify the boundaries of contributing fragments, skipping up to 93% of unnecessary background processing.

Hardware Design and Integration

The GBU hardware plug-in implements a row-centric tile engine. Unlike the SIMT model, the GBU assigns individual rows to independent Processing Elements (PEs), eliminating the idle time caused by workload imbalance. To manage memory bandwidth, the GBU features a Gaussian Reuse Cache. Leveraging the deterministic nature of tile-based rendering, the hardware employs a precomputed cache replacement policy that keeps frequently accessed Gaussian data on-chip, reducing DRAM access by 44.9%.

The GBU is designed to work alongside existing edge GPUs. The GPU handles application-specific preprocessing—such as skeletal animations for human avatars—while the GBU takes over the universal blending task, creating an efficient two-level processing pipeline.

Experimental Results

The GBU was evaluated using standard benchmarks, including the MipNeRF-360 dataset for static scenes, the Neural 3D Video dataset for dynamic scenes, and various video-based human avatar reconstructions.

Performance and Efficiency

Integrating the GBU with a Jetson Orin NX resulted in dramatic performance gains:

  • Static Scenes: Framerates increased from 13 to 92 FPS.
  • Dynamic Scenes: Framerates increased from 18 to 80 FPS.
  • Human Avatars: Framerates increased from 41 to 102 FPS.

Energy efficiency saw even more significant improvements, with a 10.8x gain for static scenes. For example, rendering 60 frames of a static scene consumed only 7 Joules with the GBU, compared to 76 Joules on the baseline GPU.

Visual Fidelity

Despite using FP16 precision to optimize hardware area and power, the GBU maintains high visual quality. The degradation in Peak Signal-to-Noise Ratio (PSNR) is less than 0.1, and the change in Learned Perceptual Image Patch Similarity (LPIPS) is less than 0.01, making the differences virtually imperceptible to the human eye.

Discussion and Limitations

While the GBU provides substantial acceleration, its efficiency is sensitive to camera distance. When the camera is far from the scene, Gaussians cover fewer pixels, reducing the opportunities for IRSS to share computations. In such cases, efficiency gains may drop from 10.8x to approximately 4.7x. Additionally, in extremely complex dynamic scenes involving multiple avatars, the preprocessing stage on the GPU may become the new bottleneck. Future research could explore extending hardware acceleration to these initial geometry and deformation phases to maintain high performance in all AR/VR scenarios.

Summary updated

BibTeX
@inproceedings{ye2025gaussian,
  author = {Zhifan Ye and Yonggan Fu and Jingqun Zhang and Leshu Li and Yongan Zhang and Sixu Li and Cheng Wan and Chenxi Wan and Chaojian Li and Sreemanth Prathipati and Yingyan Celine Lin},
  title = {{Gaussian Blending Unit: An Edge GPU Plug-in for Real-Time Gaussian-Based Rendering in AR/VR}},
  booktitle = {HPCA},
  year = {2025},
  doi = {10.1109/hpca61900.2025.00036},
}
Figure