Training-Free Acceleration of Multi-View 3D Transformers via Regime-Aware Attention Approximation

Muhammed Izzet Sezer, Sevval Belkis DikkayaarXiv 2026 · arXiv

TL;DR

RAWATTA is a training-free method that accelerates multi-view 3D transformers like VGGT-Ω by up to 2.1x while doubling the number of images that can fit on a consumer GPU. By identifying three distinct functional "regimes" in transformer layers—context, matching, and anchoring—it applies tailored precision and token-merging strategies that preserve 3D reconstruction accuracy where standard LLM-style sparsity fails.

What Problem Does This Paper Address?

Multi-view 3D transformers (e.g., VGGT, DUSt3R) have revolutionized 3D reconstruction by predicting camera poses and geometry directly from unposed images in a single forward pass. However, they rely on global attention, where every patch in every image attends to every other patch. This leads to a computational cost that grows quadratically with the number of images ($N^2$), quickly exhausting GPU memory.

While the Large Language Model (LLM) community has developed many "accelerators" (like sparse attention or token merging), these often fail for 3D tasks. In LLMs, a token might only need to attend to a few "important" words. In 3D vision, finding precise correspondences between images requires dense, high-precision matching across many views. Standard shortcuts either destroy these correspondences, leading to massive accuracy drops, or provide negligible speedups.

Methods

The authors introduce RAWATTA (Regime-AWare ATTention Approximation), which optimizes models without any retraining or fine-tuning. Their approach consists of three main components:

  1. Regime Analysis: They analyze the attention patterns of every layer across multiple datasets and identify three distinct roles (shown in Figure 1a):
  2. Diffuse Context: Early layers that average information across the entire scene.
  3. Selective Matching: Middle layers that perform dense point-to-point matching across images.
  4. Reference Anchoring: Late layers that attend heavily to the first "reference" image to establish a coordinate system.
  5. Tailored Operators: Instead of a one-size-fits-all approximation, RAWATTA applies different strategies per regime. For example, it uses token merging in context and anchoring layers but keeps matching layers dense to preserve fine details.
  6. Mixed-Precision CUDA Engine: They developed a custom CUDA engine that runs attention in 8-bit (INT8/FP8) by default, selectively reverting to 16-bit only for layers that fail a "worst-case" calibration check.
The three regimes of global attention (a) and performance vs. latency (b, c).

Datasets

The authors evaluated their method on seven diverse 3D datasets to ensure the identified regimes were stable and not overfit to specific scenes:

  • ETH3D (High-resolution multi-view)
  • CO3Dv2 (Common Objects in 3D)
  • RealEstate10K (Large-scale camera trajectories)
  • NRGBD & TUM-dynamic (RGB-D sequences)
  • 7-Scenes (Indoor localization)
  • Mip-NeRF 360 (Object-centric captures)

Key Insights and Ideas

  • Attention isn't sparse, it's structured: The "aha moment" was realizing that while 3D attention isn't sparse (tokens attend to many images), it follows a predictable functional map. Figure 2 shows how entropy and "mass" on the reference frame clearly delineate these regimes.
  • The "First Frame" is an Anchor: In models like VGGT-Ω, late layers send 50-60% of their attention mass to the very first image. This "anchoring" is essential because the model outputs coordinates relative to that first frame.
  • Matching Layers are the Bottleneck: Matching layers are extremely sensitive. Merging even a few tokens in these layers causes accuracy to collapse, explaining why previous general-purpose accelerators failed.
  • Calibration over Heuristics: Rather than manually picking layers to optimize, RAWATTA uses a "threshold schedule" and a quick calibration pass on a few frames to automatically decide which layers can tolerate approximation.
Detailed per-block analysis showing entropy (black) and mass on the reference frame (red), defining the regimes.

Key Results

  • Speed & Memory: On an NVIDIA RTX 4070 SUPER (12 GB), RAWATTA achieved a 2.08x speedup over the official VGGT-Ω code. Crucially, it allowed the model to process 200 images in a single pass, whereas the original code ran out of memory at 100 images (see Figure 6).
  • Accuracy: The "Fast" setting loses at most ~1 point of pose accuracy (AUC@3) on the worst-performing dataset, a negligible trade-off for the massive gains in throughput.
  • Training-Free: Unlike many competitors that require expensive distillation or fine-tuning, RAWATTA works "out of the box" with released model weights.
Latency vs. number of frames. RAWATTA scales significantly better than the official implementation.

Limitations and Future Work

  • Per-Model Calibration: While the regimes are stable across datasets, the specific thresholds for which layers can be approximated vary between different model architectures (e.g., VGGT-Ω vs. VGGT-1B). Each new model requires a one-time, 49-configuration search (taking a few minutes).
  • Hardware Specificity: The current speedups are heavily tied to the authors' custom CUDA kernels optimized for NVIDIA Ada Lovelace (RTX 40-series) and Blackwell architectures.
  • Future Directions: The authors suggest that future work could explore more sophisticated "top-k" routing specifically for the matching layers, which remain the most computationally expensive part of the pipeline.

Summary updated

Abstract

Multi-view 3D transformers such as VGGT reconstruct a scene from a set of unposed images in a single forward pass, predicting camera poses, depth and 3D points, but the cost of their global attention grows quadratically with the num- ber of images. Accelerators borrowed from large language models assume that each token attends to only a few oth- ers; on these models they lose accuracy or gain little speed. We study the global attention of three multi-view transform- ers on seven datasets and find that every layer plays one of three roles: it averages over the whole scene, matches points across many images, or anchors on the first image, whose coordinates the outputs use. Building on this anal- ysis, we propose Regime-AWare ATTention Approximation (RAWATTA), a training-free method that runs attention in 8- bit precision and keeps 16-bit precision only in the layers where a worst-case calibration check requires it. Its fast set- ting, RAWATTA-Fast, also approximates attention in a few averaging and anchoring layers, while matching layers and the first image keep all their keys. On a 12 GB GPU at 50 im- ages, RAWATTA runs VGGT faster than the released code, mostly through our CUDA engine, and loses at most about one point of pose accuracy (AUC@3) on any dataset; RAWATTA-Fast is slightly faster and less accurate. Both fit 200 images, where the released code runs out of memory at 100.
BibTeX
@misc{sezer2026training,
  author = {Muhammed Izzet Sezer and Sevval Belkis Dikkaya},
  title = {{Training-Free Acceleration of Multi-View 3D Transformers via Regime-Aware Attention Approximation}},
  year = {2026},
  eprint = {2611.17629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17629},
}
Figure