Large language models do not use all of their attention heads the same way. A small minority, called retrieval heads, reach far back into the context to fetch specific tokens. The large majority look only at nearby tokens. Compression methods exploit this split by giving the few retrieval heads full memory and the rest a short window, and they lose almost no accuracy. Transformers for 3D point clouds (unordered sets of 3D coordinates from LiDAR or depth scans) face the same tension between cost and context. Every head either looks at a small window, which leaves it blind to scene layout, or at a large one, which costs quadratic compute. Yet no one has asked whether a trained point transformer has already split its heads into a few long-range specialists and many local filters.
We propose to find these heads and put them to use. We will measure, head by head, how much each attention head in a pretrained serialized point transformer depends on points that are far away. The measurement uses a test that accounts for the uneven density of point clouds. We then convert the model so that only the identified heads get a much larger attention window, and the rest shrink to a small one, at equal or lower compute. The question is whether the global/local split that 3D architectures currently hand-engineer as separate branches already exists inside ordinary multi-head attention, and can be recovered cheaply from a model that has already been trained.
Core Claim
In Point Transformer V3 (PTv3), attention is computed inside windows of 1024 points. These windows follow a space-filling curve, a path that visits 3D space in an order that roughly preserves locality, so an unordered set becomes a 1D sequence. PTv3 treats all heads identically and alternates the curve between stages. We claim three things about this model.
First, a small fraction of its heads account for most of the model's dependence on distant points, once "distant" is measured relative to local sampling density rather than in raw meters.
Second, these non-local heads come in two distinct kinds:
- "seam heads" reach points that are close in 3D but far along the serialization curve, repairing the curve's discontinuities;
- "long-range heads" reach points that are genuinely far away in the scene.
Third, giving only the long-range heads a much larger window, and shrinking the others to a small window plus a summary token, matches or beats the uniform model at matched attention compute. It does so after only light fine-tuning.
The novelty relative to the closest work is specific. Retrieval Heads Meet Vision found that about 1.7–2.6% of heads in 2D vision-language models are causally responsible for grounding, and that 5–10 calibration examples suffice to find them. Its setting is regular pixel grids inside a language backbone. It neither studies native 3D backbones nor uses the heads to reallocate compute. DuoAttention and RazorAttention do reallocate memory by head. Their notion of "far" is sequence position, which is meaningless for an orderless, unevenly dense point set.
Why irregularity changes the identification problem
In text, "distant" means many positions back. In a point cloud, a fixed window of 1024 serialized points covers a few centimeters on a dense tabletop and several meters in a sparse corridor or a distant LiDAR return. A head that looks "long-range" in meters may simply be looking at its only available neighbors.
We therefore define distance relative to local density. For query point $i$, let $\rho_i$ be the distance to its $k$-th nearest neighbor, a proxy for local point spacing. Keys farther than $c\,\rho_i$ count as distant. For head $h$ with output $o_h(i)$ on the full window, we compute an ablated output $\tilde o_h(i)$. Here distant keys are replaced by a single compensation token, meaning their mean key and value, following RazorAttention's recipe. The head's retrieval score is
$$ s_h = \mathbb{E}_i \frac{\lVert o_h(i) - \tilde o_h(i) \rVert}{\lVert o_h(i) \rVert}, $$
averaged over query points in a handful of calibration scenes.
We also compute a curve-distance version of the same score, which marks keys as distant by their gap along the serialization order. Comparing the two scores separates seam heads (high curve score, low density-relative score) from long-range heads (high on both).
DuoAttention warns that raw attention weights can mislead. So the output-deviation score is only the screening step. Heads are confirmed by causal ablation of their effect on final segmentation, following the protocol of Park et al.: mask the top heads versus an equal number of random heads.
Research Program
The work has three stages.
Stage 1: analysis. On a pretrained PTv3, we compute both scores for every head at every stage, using about 10 calibration scenes. We test whether the score distribution is heavy-tailed, as in text and 2D models, or roughly uniform. We check whether the ranking is stable across calibration sets and across ScanNet versus S3DIS. We also check where the high scorers sit relative to the shuffle-order mechanism.
A negative result here is informative. If 3D heads do not specialize, then hand-built splits like APPT are necessary rather than a workaround, which settles Gap 2 of the literature in the other direction.
Stage 2: conversion. We regroup heads contiguously, as DuoAttention does.
- Long-range heads get an enlarged serialized window (4096–16384 points, or the whole scene for moderate scenes, using chunked FlashAttention).
- Local heads shrink to a small window (64–128 points) plus one compensation token that summarizes the rest of the original window.
- Seam heads keep a normal-size window, but their window is taken from a second serialization order, because their job is repairing curve seams rather than seeing farther.
We then fine-tune briefly, for a few epochs, with the head assignment frozen. Optionally we also tune DuoAttention-style gates, one learnable scalar per head.
Stage 3: comparison against training from scratch. We train the same asymmetric architecture from scratch with a designated split. This tells us what share of the benefit comes from the architecture itself and what share from finding the right heads in an existing model.
Baselines
The unmodified pretrained PTv3. This is the do-nothing reference. It is also the strongest existing method on this question, since it is the best attention backbone in the file and has the largest window. PTv3 reports 79.4% mIoU (mean intersection-over-union) on ScanNet v2 semantic segmentation when pre-trained, along with a 3.3× speedup and 10.2× memory reduction over PTv2.
We will rerun the released checkpoint on our evaluation pipeline and use our reproduced number as the reference, reporting both.
For S3DIS Area 5, no reported PTv3 result was found on Lacuna. The published comparison points there are:
- APPT: 72.3% mIoU at 126G FLOPs;
- Stratified Transformer: 72.0% mIoU at 280G FLOPs;
- PTv2: 71.6% mIoU.
Beating PTv3 means one of two outcomes, averaged over at least three fine-tuning seeds and larger than seed variance:
- equal attention FLOPs and memory, with higher mIoU;
- matching mIoU (within 0.3 points) at substantially lower attention memory.
The hand-engineered global/local splits. These are APPT and Stratified Transformer. APPT uses separate local and pivot-based global attention branches. Stratified Transformer makes every head attend to both dense local keys and sparse distant keys. They represent the alternative answer to the same question, namely building the split in rather than finding it. We compare against their reported S3DIS numbers and FLOPs. Where we can, we also retrain them on our data pipeline with the same epoch budget.
Two ablations that isolate identification. These matter most.
Random assignment. The same asymmetric conversion, with the same number of wide heads per stage, but heads chosen at random. It gets the same fine-tuning budget and learning-rate sweep.
Worst-case assignment. The same conversion with the lowest-scoring heads made wide.
If random assignment plus fine-tuning matches identified assignment, then identification buys nothing in this regime and the claim fails.
A further ablation removes the density normalization, scoring by raw metric distance. Another merges seam heads into the long-range group. Together these test whether the irregularity-aware parts of the method matter.
Why not just…?
Why not just enlarge PTv3's window for every head? This is the most direct alternative. Attention cost within serialized windows grows linearly with window size per point, so a uniform 4096-point window costs roughly four times the attention FLOPs of the 1024 default. Our comparison runs uniform windows at the same total attention budget as the asymmetric model: for example, a uniform window around 1500 points against 25% of heads at 4096 and 75% at 128. If uniform widening wins at matched cost, the head-specialization premise is wrong for 3D and we will report that. The text-model evidence that about 15–25% of heads carry long-range retrieval (RazorAttention, DuoAttention) motivates the bet but does not guarantee it transfers.
Why not just assign any 25% of heads as global and fine-tune? This is a real threat. Networks can adapt heads to whatever role the architecture imposes, and from-scratch training certainly will. That is exactly why the random-assignment ablation and the from-scratch designated-split run are included. The proposal's claim is deliberately narrow: identification matters when converting an expensive pretrained model under a small fine-tuning budget. We will sweep the fine-tuning length and report where, if anywhere, random assignment catches up. If it catches up within one or two epochs, identification is a convenience rather than a contribution.
Why not just use a dual-branch design like APPT or Stratified Transformer? Those designs work: 72.3% and 72.0% on S3DIS. However, they need purpose-built modules. APPT uses farthest-point sampling for pivots and a hand-tuned capacity ramp across depth. Stratified Transformer pays 280G FLOPs and needs custom CUDA kernels. Neither can be applied to an existing pretrained serialized backbone. Our comparison asks whether a converted monolithic model reaches their accuracy at lower cost. If it does not, their explicit split remains the better engineering choice.
Why not just switch to a linear-time state-space model such as PointMamba? These models remove quadratic cost, but they compress history into a fixed-size state along a 1D traversal. RazorAttention's analysis ties exact retrieval to query-key matching, which a fixed state approximates poorly. We found no reported ScanNet or S3DIS segmentation result on Lacuna for the Mamba-based point models in the file. We therefore treat them as a secondary comparison, run under the same training budget if compute allows, rather than as the main baseline.
Why not just downsample points? Point Transformer documents that aggressive downsampling discards the fine geometry that segmentation needs. Our method keeps every point and changes only which heads see how far.
Proposed Experiments and Decisive Checks
The main benchmarks are:
- ScanNet v2 semantic segmentation (validation mIoU);
- S3DIS Area 5 mIoU, with per-class IoU reported;
- attention FLOPs, peak memory, and latency on the same GPU.
Beyond the headline numbers, we will look at classes whose identity depends on scene layout rather than local shape: door, window, and board against wall on S3DIS, and table against desk on ScanNet. The hypothesis predicts gains concentrated in those classes.
As a direct probe of long-range dependence, we run a context-swap test. We replace all points beyond a density-relative radius with the corresponding region from a different scene and measure how often predictions change. The identified long-range heads should carry most of that sensitivity. Masking them should reduce it far more than masking random heads.
Outdoor generalization is tested on Waymo detection with a FlatFormer-style equal-group backbone, where density falls sharply with range. FlatFormer reports only relative figures: 4.6× faster than SST, and 1.3 L2 mAPH above PillarNet at similar latency. No absolute Waymo mAPH was found on Lacuna, so we reproduce FlatFormer and report against our reproduction.
Four results decide the project, in order.
- Does specialization exist? Masking the top 5% of heads by score must hurt mIoU several times more than masking 5% of random heads, and the head ranking must be stable across two disjoint 10-scene calibration sets. If not, the project ends with a negative analysis result.
- Does density normalization matter? On outdoor data, scoring with density normalization must pick heads whose conversion yields higher accuracy than scoring by raw distance.
- Does identification matter? At matched attention compute and matched short fine-tuning, identified assignment must beat random assignment and uniform widening. It should also approach the accuracy of the from-scratch designated split at a fraction of its training cost.
- Is the seam/long-range distinction real? Re-serializing seam heads should help, while widening them should not, and the reverse should hold for long-range heads.
References
- Point Transformer V3: Simpler, Faster, Stronger
- Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- APPT : Asymmetric Parallel Point Transformer for 3D Point Cloud Understanding
- Stratified Transformer for 3D Point Cloud Segmentation
- Point Transformer V2: Grouped Vector Attention and Partition-based Pooling
- PointMamba: A Simple State Space Model for Point Cloud Analysis
- Point Transformer
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
- 3D Semantic Parsing of Large Scale Indoor Spaces
- FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer