AVGGT: Rethinking Global Attention for Accelerating VGGT

Xianbing Sun, Zhikai Zhu, Zhengyu Lou, Bo Yang, Jinyang Tang, Liqing Zhang, He Wang, Jianfu Zhang

Introduction

Foundation models for 3D vision, such as VGGT and $\pi^3$, have significantly advanced tasks like camera pose estimation and 3D reconstruction. These models typically employ an alternating architecture of frame-level attention (intra-image) and global attention (inter-image). However, the quadratic computational complexity ($O(N^2)$) of global attention relative to the number of frames $N$ creates a significant bottleneck, particularly for long video sequences exceeding 300 frames. Existing sparse attention methods often degrade performance by failing to account for the specific information processing patterns of 3D vision transformers.

The AVGGT Method

The researchers introduce AVGGT, a training-free acceleration pipeline designed to optimize 3D vision transformers without sacrificing accuracy. The approach is based on a layer-wise analysis of how these models process spatial information.

Overview of the AVGGT acceleration strategy, showing layer-wise conversion and subsampling intuition alongside performance gains.

1. Global-to-Frame Conversion in Early Layers

Analysis revealed that early transformer layers primarily focus on per-view feature extraction rather than cross-view correspondence. By converting the first approximately nine global attention layers into efficient frame-level attention, the authors significantly reduce computational overhead with negligible impact on accuracy.

2. Subsampling Global Attention in Deep Layers

In middle and late layers where cross-view alignment occurs, the authors found that dense matching is redundant. AVGGT implements a uniform grid-based subsampling of Key (K) and Value (V) tokens.

  • Query Preservation: All Query (Q) tokens are retained to ensure every pixel receives a 3D update, maintaining the density of final predictions.
  • Diagonal Preservation: Self-interaction terms are kept to maintain local feature coherence.
  • Mean-Fill Component: Dropped tokens are approximated by a mean component to preserve global attention statistics.

Experimental Evaluation

AVGGT was evaluated on several standard benchmarks, including 7-Scenes (dense multi-view), RealEstate10K, TUM-dynamics, and DTU.

Key Results

The method achieves substantial speedups: 2× for 100 frames and up to 10× for 800 frames. Crucially, AVGGT maintains or even improves accuracy in dense settings where original models might fail due to memory constraints.

Camera pose estimation results comparing original models (VGGT, $\pi^3$) with accelerated variants across time and accuracy metrics.

Conclusion

AVGGT demonstrates that 3D-aware attention pruning is a powerful, training-free way to scale vision transformers to long sequences. By recognizing the "division of labor" between layers, it provides a robust alternative to generic sparse attention mechanisms. Future work may involve adaptive split-point selection and further low-level hardware optimizations.

Summary updated

BibTeX
@inproceedings{sun2026avggt,
  author = {Xianbing Sun and Zhikai Zhu and Zhengyu Lou and Bo Yang and Jinyang Tang and Liqing Zhang and He Wang and Jianfu Zhang},
  title = {{AVGGT: Rethinking Global Attention for Accelerating VGGT}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=cabMyiBmPq},
}
Figure