GPA-VGGT: Geometry-Preserving Acceleration of VGGT

Muyeol Sung, Jangwon Kim, Jongwoo LimarXiv 2026 · arXiv

TL;DR

GPA-VGGT is a training-free acceleration method for the Visual Geometry Grounded Transformer (VGGT) that achieves a ~2.6x speedup while retaining over 99% of the original model's geometric precision. By identifying and fixing critical information loss in previous "token merging" techniques, it enables high-fidelity 3D reconstruction for long image sequences without the need for costly fine-tuning.

What Problem Does This Paper Address?

Modern 3D vision models like VGGT are incredibly powerful: they can take dozens or even hundreds of unposed images and jointly estimate camera parameters, depth maps, and 3D point clouds in a single forward pass. However, they have a major scalability problem. Their "global attention" mechanism, which allows every part of every image to look at every other part, has a computational cost that grows quadratically ($O(N^2)$) with the number of frames $N$.

While researchers have tried to speed this up using "training-free" methods like token merging (grouping similar visual features to reduce the workload), these shortcuts usually come at a steep price: a significant drop in geometric accuracy. This is especially visible under strict evaluation metrics like AUC@3 (Area Under the Curve at a 3-degree error threshold), which measures fine-grained pose precision. Previous acceleration methods might look okay under loose metrics, but they fail when high precision is required for actual 3D reconstruction.

GPA-VGGT maintains high accuracy (AUC@3) while significantly reducing runtime compared to vanilla VGGT and other baselines.

Methods

The authors propose GPA-VGGT (Geometry-Preserving Acceleration of VGGT), which introduces three key innovations to speed up the model without losing the "geometry" information:

  1. Query-Preserving K/V Merging: Previous methods merged "queries" (the tokens asking for information), which collapsed the unique details of each image patch. GPA-VGGT keeps all queries but merges the "keys" and "values" (the information sources). This ensures every patch still produces its own unique output.
  2. Token-Size Compensation: When multiple tokens are merged into one, that single token should carry more "weight" in the attention process. GPA-VGGT adds a +log s correction (where $s$ is the number of merged tokens) to the attention scores to restore this "attention mass."
  3. Pooled Cross-Frame Broadcast: In early layers, instead of full global attention, GPA-VGGT uses "frame attention" (looking only within the same image) but augments it with a compact, shared set of "summaries" from all other frames. This provides enough global context to prevent "representational drift" between views without the full quadratic cost.

To further reduce overhead, they use Gaussian Random Projection to map high-dimensional features (1024-d) down to 128-d just for the purpose of deciding which tokens to merge, and they implemented custom Triton kernels to make these operations lightning-fast on GPUs.

The GPA-VGGT architecture replaces early global blocks with frame attention + shared context and uses sensitivity-aware merging in later blocks.

Datasets

The model was evaluated on several standard 3D vision benchmarks:

  • CO3D (Common Objects in 3D): Used for evaluating camera pose accuracy on diverse object categories.
  • RealEstate10K (RE10K): Used for pose estimation in large-scale indoor/outdoor scenes.
  • 7-Scenes & NRGBD: Used to evaluate the quality of point-map reconstruction (accuracy, completeness, and normal consistency).

Key Insights and Ideas

  • Precision requires diversity: Merging queries is the primary cause of "blurry" or "doubled" geometry in prior work. By preserving all queries, GPA-VGGT maintains the high-frequency details needed for precise 3D points.
  • The "First-Frame Bias": The authors discovered that without the +log s correction, attention becomes accidentally biased toward the first frame of a sequence, leading to massive accuracy drops.
  • Early layers need a "Global Anchor": Even if early layers don't find perfect point-to-point matches across views, they need a shared geometric reference. Without the "Pooled Cross-Frame Broadcast," each frame's features drift in different directions, ruining the final pose estimate.
Analysis shows that merging queries (middle) collapses output diversity, while GPA-VGGT's approach (right) preserves it. Omitting the log s correction (bottom middle) causes a severe bias toward the first frame.

Key Results

  • Speed: GPA-VGGT processes a 200-frame sequence in 33.6 seconds, compared to 86.8 seconds for the original VGGT—a ~2.6x speedup.
  • Accuracy: On the strict CO3D AUC@3 metric, GPA-VGGT scores 56.94, nearly matching the vanilla model's 57.49. In contrast, the previous state-of-the-art acceleration (FastVGGT) drops to 41.56.
  • Visual Quality: Qualitative results show that GPA-VGGT avoids the "ghosting" and surface artifacts seen in other accelerated models, producing point clouds that are virtually indistinguishable from the full-cost original.
Qualitative comparison shows GPA-VGGT (right) produces clean, single-surface reconstructions, whereas other fast methods (middle) create noisy or doubled surfaces.

Limitations and Future Work

While GPA-VGGT is a significant step forward, the authors note a few areas for improvement:

  • Matching Overhead: Even with random projections, the process of deciding which tokens to merge still takes up about 18% of the forward pass.
  • Core-Block Selection: The method relies on identifying "core blocks" that are sensitive to merging. While they provide a data-driven way to find these, it's an extra step in the pipeline.
  • Future Directions: The authors suggest that these geometry-preserving principles could be applied to other transformer-based 3D backbones beyond VGGT, potentially leading to a new standard for efficient 3D perception.

Summary updated

Abstract

Feed-forward geometry models such as VGGT recover camera parameters, depth maps, and point maps from one to hundreds of views in a single forward pass, making them powerful backbones for multi-view 3D perception. However, VGGT's global attention is quadratic in the number of frames and dominates end-to-end runtime on long sequences, which has motivated a range of training-free efficiency methods such as token merging. These methods, however, come with a drop in accuracy that makes them hard to use in place of VGGT. We propose GPA-VGGT, a training-free acceleration method that replaces global attention in early blocks with frame attention augmented by a shared cross-frame K/V set. In the remaining blocks, we merge only keys and values while preserving all queries, restore the attention mass of the tokens each merged key represents, and randomly project the matching metric to 128 dimensions to reduce matching cost. Across diverse multi-view geometry tasks, GPA-VGGT provides training-free acceleration at speedups comparable to prior work while preserving the information required for precise geometric estimation, retaining over 99% of VGGT's pose accuracy under AUC@3.
BibTeX
@misc{sung2026gpa,
  author = {Muyeol Sung and Jangwon Kim and Jongwoo Lim},
  title = {{GPA-VGGT: Geometry-Preserving Acceleration of VGGT}},
  year = {2026},
  eprint = {2610.37000},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37000},
}
Figure