TL;DR
Cross3DGS is a unified, Vulkan-based trainer that brings high-quality 3D Gaussian Splatting (3DGS) training directly to mobile devices. By using a clever two-step "schedule selection" process to navigate hardware limits and driver quirks, it achieves workstation-level reconstruction quality across diverse GPUs from NVIDIA, AMD, Intel, Arm, Qualcomm, and Apple.
What Problem Does This Paper Address?
3D Gaussian Splatting has revolutionized real-time radiance field rendering, but the training process remains largely tethered to high-end CUDA-enabled workstations. While we can capture stunning scenes with our phones, we usually have to ship that data to a server or a beefy PC to turn it into a 3D model.
The challenge in moving training to mobile devices isn't just about translating code from CUDA to a portable language like Vulkan. Mobile GPUs are a "wild west" of varying constraints: different memory limits, varying subgroup widths, and—perhaps most frustratingly—unreliable drivers that might report a feature as supported only to crash when it's actually used. Existing portable trainers often fail on mobile hardware because they assume a level of hardware uniformity that simply doesn't exist.
Methods
The authors introduce Cross3DGS, a cross-vendor, cross-platform trainer built on Vulkan 1.1. The core innovation is decoupling the mathematical optimizer from its physical execution on the GPU.
Instead of a single "one-size-fits-all" kernel, Cross3DGS maintains a catalog of 256 execution schedules. These schedules represent different ways to perform the same 3DGS training steps (like projection, sorting, and gradient calculation) by varying parallel work layouts and memory strategies.
To decide which schedule to use for a specific device, they employ a two-step selection process:
- Feasibility Rules: A set of 13 rules filters the catalog. These rules check device limits (e.g., shared memory size), workload requirements (e.g., image resolution), and—crucially—driver rules that exclude schedules known to fail on specific hardware like Qualcomm's Adreno GPUs.
- Preference Order: From the remaining feasible schedules, the system picks the best one based on a fixed preference order determined by offline benchmarking. This avoids the need for slow on-device timing tests.
Datasets
The researchers evaluated Cross3DGS using a mix of standard benchmarks and new real-world captures:
- Standard Benchmarks: Mip-NeRF 360 (7 scenes), Tanks and Temples (Train, Truck), and Deep Blending (DrJohnson, Playroom).
- Mobile Captures: New handheld captures of indoor and outdoor objects (plush toy, shrine, statue) used to demonstrate the end-to-end mobile application.
Key Insights and Ideas
- Portability is an Execution Problem: Writing "portable" kernels in a shading language isn't enough. True portability requires managing the schedule of how those kernels interact with memory and command queues.
- Driver Bugs are First-Class Constraints: The authors explicitly acknowledge that hardware specs are often aspirational. By encoding known driver failures into their feasibility rules, they achieve a level of reliability that "spec-compliant" code cannot.
- Numerical Consistency is King: To ensure that a model trained on an iPhone looks the same as one trained on an RTX 6000, they use a CPU-based double-precision reference to verify that every feasible schedule produces the same parameter updates within a tight tolerance.
Key Results
- Unmatched Coverage: Cross3DGS successfully completed standard-quality training (30k updates, 1M Gaussians) on all 8 tested configurations, including Android phones and an iPhone. Baselines like VkSplat and Brush failed on several mobile GPUs.
- Identical Quality: The final reconstruction quality (PSNR) was nearly identical across all devices, staying within a tiny 0.07 dB margin.
- Workstation Competitive: On high-end hardware, Cross3DGS delivered higher quality than VkSplat and was roughly 3x faster than Brush at comparable quality levels.
- True Mobile Autonomy: The authors demonstrated a mobile app that handles the entire pipeline—capture, sparse Structure-from-Motion (SfM), 3DGS optimization, and interactive viewing—entirely on-device.
Limitations and Future Work
- Memory Footprint: To maximize speed, Cross3DGS keeps its buffers resident in GPU memory, which results in a higher peak memory usage (approx. 2.7 GB for a standard run) compared to some baselines.
- Fixed Preferences: The current system uses a fixed preference order. While effective, a one-time "timing screen" at first launch could potentially find even faster schedules for specific driver versions.
- Extreme Mobile: The authors noted that while phones and tablets are now supported, even more constrained devices like AR glasses or headsets remain a target for future optimization.
Summary updated