Vision transformers decide which image patches should exchange information by comparing their features. Every patch is compared with every other patch, so the cost grows with the square of the number of patches. "Linear attention" removes that quadratic cost by rewriting the comparison as a product of per-patch feature maps. This rewrite has a side effect: it makes it hard to tell the model that two patches are physically far apart. On a road scene, a pedestrian's silhouette and the wall twenty meters behind it sit next to each other on the image grid. A linear-attention model has no cheap way to stop them from mixing.
We propose a linear-time attention layer whose weights decay with 3D metric distance between the points that two patches see, rather than their distance on the image grid. The layer also averages that distance term over each patch's uncertainty in depth along its camera ray. The construction rests on one observation. A Gaussian kernel on 3D distance can be written exactly as an expected inner product of positive random features. Taking the expectation of those features under a Gaussian depth error along the viewing ray has a closed form, so depth uncertainty can be handled exactly in expectation without breaking linearity. The result is attention that is linear in the number of patches, sharp across real depth discontinuities, and soft where depth is unreliable. Unreliable depth is exactly where existing 3D positional encodings smear features.
Core Claim
Two facts from the literature motivate this work. First, the useful ingredient in image-based 3D perception is lifting pixels to metric coordinates $(X,Y,Z)=d\,K^{-1}[u,v,1]^T$, where $(u,v)$ is the pixel, $d$ its depth, and $K$ the camera intrinsics. The specific 3D data structure matters less: Rethinking Pseudo-LiDAR Representation shows that feeding these coordinates as image channels to a 2D network matches point-cloud networks. Calibrated lifting also keeps foreground and background apart at object boundaries (Unsupervised Depth Completion with Calibrated Backprojection Layers). Second, the one widely cited monocular detector that combines depth-aware encoding with linear attention, MonoDTR, only adds learned embeddings of discrete depth bins to the features. It then runs generic linear attention over the 2D tokens. Its attention never computes anything metric, and it reports both depth-error-driven feature smearing and weak results on small, non-rigid classes.
Our claim is that a factorized 3D distance kernel, averaged over depth uncertainty, can be placed inside linear attention at linear cost. We also claim it fixes the specific failure MonoDTR leaves open: mixing across depth discontinuities, which hurts most on pedestrians, cyclists, and distant objects.
Concretely, for query token $i$ and key token $j$ the unnormalized weight is
$$ w_{ij} = \phi_c(q_i)^T\phi_c(k_j)\cdot \mathbb{E}\big[\exp\!\big(-\|X_i - X_j\|^2 / 2\ell^2\big)\big], $$
where $\phi_c$ is a positive content feature map as in standard linear attention, $X_i$ is the random 3D point seen by token $i$, $\ell$ is a learned per-head length scale in meters, and the expectation is over independent depth errors of the two tokens. We write the Gaussian kernel with positive random features $\psi_\omega(X)=\exp(\omega^T X/\ell - \|X\|^2/2\ell^2 - \|\omega\|^2/2)$ with $\omega\sim\mathcal{N}(0,I_3)$. We model token $i$'s position as Gaussian with mean $\mu_i$ and covariance $\Sigma_i=\sigma_i^2 r_i r_i^T$, where $r_i$ is the unit ray direction and $\sigma_i$ comes from the depth head's predicted distribution. Under that model, the expectation of each feature is available in closed form, a Gaussian moment generating function evaluated along the ray. Because the two tokens' errors are independent, the product of expected features gives the expected kernel exactly. The weight therefore factors into a per-query vector and a per-key vector, the tensor product of the content features and the expected geometric features. The usual associative trick applies, and attention costs $O(N\,d_c\,m\,d_v)$, where $N$ is the token count, $d_c$ the content feature width, $m$ the number of 3D features, and $d_v$ the value width. Three dimensions is low enough that $m$ can be small. We will also test deterministic 3D Gauss–Hermite quadrature nodes in place of random $\omega$, in the spirit of the quadrature-based positive features in SLAY.
The closest prior pieces each do part of this. GTA applies $SE(3)$ group transforms token by token so that the geometric overhead is linear, but the attention it feeds is still quadratic softmax, and the transforms encode relative pose, not a decaying metric locality. SLAY's "geometry" is the hypersphere of the embedding space and never touches scene coordinates. M-LRM restricts cross-attention to ray intersections but keeps dense quadratic attention in half its layers and is fragile to camera errors. Voxel-based Dynamic Token Compression uses 3D lifting only as an offline, training-free token-merging step and needs sensor depth. Our layer is a trainable attention operator that uses estimated depth and its uncertainty directly.
Research Program
We will build the layer as a drop-in replacement for the linear attention in MonoDTR's depth-aware transformer. That backbone is chosen because it already has a depth-distribution head, trained with LiDAR supervision that is discarded at test time, and it already uses linear attention. Swapping only the attention operator isolates our ingredient. The depth head's per-pixel categorical distribution gives $\mu_i$ through its mean and $\sigma_i$ through its spread, so no new supervision is needed.
The analysis has three parts. First, we will measure approximation quality: how closely the factorized kernel matches the exact expected Gaussian kernel as $m$ varies, including far-apart pairs, where positive random features have high variance. Second, we will study what the learned length scales $\ell$ settle to per head and per layer. We expect some heads to become near-global, with large $\ell$, and others to become boundary-sensitive local heads, which would explain why no dense layers are needed. Third, we will run a controlled resolution sweep to confirm linear memory and latency scaling against a softmax layer carrying the same 3D bias.
As a secondary, smaller check of generality, we will substitute the layer for intra-view attention in a reduced-scale multi-view synthesis model. Intra-view attention is the part that Efficient-LVSM leaves quadratic. Running this check at Efficient-LVSM's full training scale is beyond our budget, so it is explicitly a smaller-scale probe.
Baselines
The naive reference point is MonoDTR unchanged: discrete-bin depth positional encoding plus generic linear attention, retrained with our code, schedule, and seeds. It reports Car Moderate $AP_{3D}$ of 15.39% on the KITTI test set at 37 ms on a V100 (MonoDTR). It is the right floor because it differs from our model only in the attention operator.
The strongest reported competitor in the file on this benchmark is MonoDETR trained with attention-based depth distillation, ADD. It reports KITTI test Car $AP_{3D}$ of 25.61 / 16.81 / 13.79% on Easy / Moderate / Hard. It is the strongest because it already injects ground-truth-depth 3D encodings into attention during training at no inference cost. We will compare against its published numbers and, on the validation split, against a retrained run with matched epochs and augmentation. Beating it means exceeding 16.81% Moderate on the test server, or matching it while using linear attention at higher input resolution. Separately and more importantly, it means a clear validation gain on Pedestrian and Cyclist and on distance-binned Car AP beyond 40 m.
A third baseline isolates whether linearization costs anything. This is quadratic softmax attention with the same expected 3D Gaussian bias added in closed form, at native KITTI resolution where it still fits in memory. It is an upper reference, not a competitor. The key ablations, all with identical data and tuning budget, are four: the 3D kernel with uncertainty set to zero (point depth), the kernel computed on 2D pixel coordinates instead of 3D, continuous sinusoidal $(X,Y,Z)$ encodings simply added to MonoDTR's tokens, and a 3D rotary encoding applied before linear attention. Hyperparameter search gets the same number of trials for every variant.
Why not just…?
Why not just add continuous 3D sinusoidal coordinates to the tokens of an existing linear transformer? This is the cheapest option, and sinusoidal 3D encodings work well in Video-3D LLM under softmax attention. Under linear attention, though, an additive encoding only shifts the content features. The kernel has no term that shrinks with $\|X_i-X_j\|$, so nothing forces suppression across a depth edge. The point-depth and sinusoidal-addition ablations test this directly. If additive encodings close most of the gap on pedestrians and far objects, our mechanism is not needed, and we will report that.
Why not just use GTA-style or rotary 3D transforms followed by linear attention? This is the most serious threat, because those transforms are token-wise and therefore compatible with linear attention. They encode relative geometry as rotations, which produce oscillating rather than decaying affinities, and their inner products are not guaranteed non-negative. That conflicts with the normalization problems SLAY documents for linear kernels, whose Performer and Cosformer baselines reach validation losses of 5.45 and 5.20 against softmax's 4.64. They also have no notion of depth uncertainty. The 3D-rotary ablation settles this. If it matches us, the contribution reduces to the uncertainty marginalization, which the point-depth ablation measures separately.
Why not just keep softmax and use windows, BEV grids, or offline voxel merging? Windowed attention loses the global context MonoDTR's transformer is meant to provide, and M-LRM found it still needed dense layers for global interaction. Voxel merging as in DTC requires sensor depth and sits outside the network. At KITTI resolution, quadratic softmax with a 3D bias may well be affordable. That is why it appears as an upper reference, and why our efficiency claim rests on the resolution sweep rather than on native-resolution latency.
Why not just distill 3D knowledge into the student, as ADD does? Distillation improves what the student attends to but leaves its cost unchanged, so it does not address scaling to higher resolution. It is also orthogonal to our layer: we will additionally report ADD applied on top of our model to show whether the gains add up.
Proposed Experiments
The primary benchmark is KITTI monocular 3D detection, reported as $AP_{3D}$ at the standard IoU thresholds for Car, Pedestrian, and Cyclist across difficulty levels. It is supplemented by validation AP binned by ground-truth distance and by a boundary metric: depth error within a few pixels of instance edges, measured on validation frames with LiDAR. The resolution sweep reports peak memory and latency from native resolution up to 2–3× scale on one GPU. The comparison is against the softmax-with-3D-bias reference, which should run out of memory first, and against MonoDTR's linear attention, which should match our scaling while losing accuracy.
The decisive checks are three. Our layer must beat MonoDTR and the sinusoidal and rotary ablations on Pedestrian, Cyclist, and far-Car AP, not only on overall Car Moderate. The uncertainty-marginalized kernel must beat the point-depth kernel most on frames where the depth head is least confident, which we will stratify by predicted $\sigma$. The gap to the softmax-with-3D-bias reference must stay within run-to-run noise at native resolution.
For the secondary multi-view probe on RealEstate10K, the reference figures are 22.85 dB PSNR for GTA's reduced-scale setup (GTA) and 29.86 dB for Efficient-LVSM at full scale. We will compare only against a same-scale retrained Efficient-LVSM variant with quadratic intra-view attention, and we will not claim parity with the full-scale figure.
References
- Rethinking Pseudo-LiDAR Representation
- Unsupervised Depth Completion with Calibrated Backprojection Layers
- MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer
- SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernel
- GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers
- Multi-View Large Reconstruction Model via Geometry-Aware Positional Encoding and Attention
- Zero-shot 3D Question Answering via Voxel-based Dynamic Token Compression
- Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention
- Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding