Retuning the Geometry, Not the Features: Low-Rank Spectrum Adaptation for 3D Rotary Transformers

Many 3D models are Transformers that read coordinates through rotary position encoding (3D RoPE). In 3D RoPE, each query and key vector is rotated by angles proportional to a point's $x$, $y$ and $z$ coordinates, so that attention between two points depends on how far apart they are and in which direction. The set of rotation speeds, called frequencies, fixes which distances the model can "see." That set was tuned to the scale and density of the pretraining data. Move the model from indoor rooms to deforming animated characters, or from rooms to outdoor LiDAR sweeps, and the frequencies no longer match the geometry.

The standard cheap fix is low-rank adaptation (LoRA), which adds a small trainable low-rank matrix to each frozen weight matrix. We point out a structural blind spot in it. LoRA on the query and key projections changes the vectors that get rotated, but it cannot change the rotation frequencies themselves. So it can only re-route information among the distance bands the model already has. We propose adapting the positional pathway directly with a tiny, structured update to the frequency matrix: a per-layer 3×3 coordinate map plus a per-frequency scale. The update keeps attention a function of relative position only, adds zero inference cost, and can be combined with ordinary LoRA. The claim to test is that when the geometry shifts, this handful of parameters recovers most of what full fine-tuning recovers, and that feature LoRA needs rank that grows with the size of the scale mismatch to do the same.

Core Claim

In a 3D RoPE layer, the attention logit between points $i$ and $j$ is $x_i^\top W_q^\top R\big(\Omega (p_j - p_i)\big) W_k x_j$. Here $x$ are token features, $W_q, W_k$ are the query and key projections, $p \in \mathbb{R}^3$ are coordinates, $\Omega \in \mathbb{R}^{(d/2)\times 3}$ stacks one frequency vector $\omega_m$ per rotation plane, and $R(\cdot)$ is the block-diagonal rotation. In the axis-aligned 3D RoPE used by Lepard and SpaceFormer, each $\omega_m$ points along a single axis, and the frequencies form a fixed geometric ladder.

LoRA on $W_q$ and $W_k$ changes which planes carry signal. It never changes $\Omega$. We adapt $\Omega$ per layer $l$ as

$$\omega'_{l,m} = e^{s_{l,m}}\, (I + U_l V_l^\top)^\top \omega_{l,m},$$

where $s_{l,m}$ is a learned log-scale for plane $m$, and $U_l, V_l \in \mathbb{R}^{3\times r}$ with $r \le 3$ give a low-rank coordinate map covering rescaling, anisotropic stretch and rotation of the coordinate frame. Both are initialized to zero, so training starts from the pretrained model. The angle stays linear in position, so relative-position dependence is preserved exactly.

Positions still enter only through the rotation. This keeps the feature–coordinate separation that Lepard found essential for avoiding overfitting to absolute coordinates. Because the new frequencies are precomputed constants, the update folds away at deployment.

The claim has a sharp prediction. The pretrained ladder is geometric, so rescaling coordinates by a factor $\kappa$ roughly shifts band $m$ onto band $m + \log_b \kappa$, where $b$ is the ladder's ratio. For feature LoRA to emulate this, it must permute channels across many planes, and a permutation of that kind is high-rank. So the LoRA rank needed to match spectrum adaptation should grow with the scale mismatch. Spectrum adaptation should need no extra capacity.

Position Relative to Prior Work

Three lines of work supply the motivation and the method.

RoSA supplies the closest precedent. It shows that 1D RoPE makes low-frequency dimensions carry disproportionately large activations, and it restricts LoRA to those query/key dimensions. It reports 85.9% commonsense micro-average on Qwen2.5-7B, against 84.3% for LoRA. RoSA still adapts only projections around fixed frequencies, and only for 1D text. We adapt the frequencies themselves, along three continuous axes.

ID-Align motivates the scale problem. It shows RoPE's decay with distance breaks attention when position indices inflate, and it repairs this by training-free remapping of 2D grid IDs. We make the remapping learnable and continuous rather than rounded to integers.

Point Transformer V2 shows that learning how strongly position conditions attention matters. It trains a position multiplier from scratch. We learn an analogous knob as a post-hoc adapter on a frozen model.

The 3D fine-tuning literature constrains the design. Point-PEFT adds point-prior prompts and neighbor-aggregating adapters. GEM adds 3D-convolution and latent-token adapters for scene segmentation. PointGST tunes in a graph spectral domain, which requires eigendecomposition. All three add coordinate-dependent residual features to the frozen hidden states. None touches how coordinates enter attention.

GEM also documents a persistent failure transferring indoor models to SemanticKITTI, which is exactly the geometric shift we target. Positional Prompt Tuning is the nearest "tune the positional module" work. It retrains an additive input positional MLP and still needs in-block adapters. It does not address rotary encodings, whose positional effect lives inside every attention layer.

LoRA3D applies rank-16 LoRA to a 2D-patch geometric model and uses no 3D positional encoding, so it is not a competitor on this mechanism.

Research Program

We first build the adapter for two 3D RoPE backbones. The primary one is Lepard, which uses 3D RoPE natively and has public training setups on the 3DMatch and 4DMatch families. The secondary one is a 3D RoPE point Transformer that we pretrain for indoor scene segmentation on ScanNet v2. The file lists no pretrained 3D RoPE scene backbone, so building one is a cost we accept.

Next we run a diagnostic analysis before any accuracy claims. For each layer and head we measure the attention-versus-distance profile in source and target domains, and ask whether the target optimum is a shifted spectrum. We then fit the LoRA rank needed to match spectrum adaptation as a function of a controlled coordinate scale factor $\kappa$. The same measurement tells us which bands carry energy, a 3D analogue of RoSA's frequency finding, and whether that energy moves under the shift.

Finally, we study the combined adapter: spectrum adaptation plus low-rank query/key updates restricted to the planes the diagnostic flags as active. This is the configuration we expect to win overall.

Baselines

The naive reference is the frozen source model evaluated zero-shot on the target domain. It shows how large the geometric gap is.

The simplest adaptive reference is a single global coordinate rescale. One scalar multiplies all input coordinates, tuned on target validation data by grid search, with a per-axis variant as well. This is our method with $s$ fixed and $U_l V_l^\top$ forced to one scalar shared across layers. It is the baseline most likely to embarrass the proposal.

The strongest parameter-efficient competitor on the feature side is vanilla LoRA on $W_q, W_k, W_v, W_o$. Published 3D numbers put it behind geometric adapters: 86.3% versus 89.1% on ScanObjectNN in Point-PEFT, and 44.2 versus 46.6 mIoU on ScanNet++ in GEM. We sweep its rank from 4 to 64, so it receives at least as many parameters as our method at every point on the curve.

The strongest existing 3D method is GEM, which reports 46.6 mIoU on ScanNet++ against 43.7 for full fine-tuning. That number is on GEM's own 124M backbone, not ours. We will reimplement GEM's spatial and context adapters on our backbone rather than compare across architectures.

Full fine-tuning is the ceiling reference. On registration, an in-domain reference is Lepard trained directly on 4DMatch, which reaches 66.9% non-rigid feature matching recall on 4DLoMatch per the Lepard paper. We will also retrain it ourselves.

The key ablation separates the two parts of the adapter: per-band scales only, coordinate map only, and both. The coordinate-map-only variant at one shared layer reduces to global rescaling, so the ablation directly measures what the per-layer, per-band structure adds.

All methods get the same target data, the same epoch budget, and the same number of learning-rate and hyperparameter trials. Results are reported over three seeds.

Beating the baselines means two things. First, spectrum adaptation plus restricted LoRA must exceed the best vanilla LoRA and reimplemented GEM at equal or smaller parameter count, with no inference overhead. Second, it must beat the tuned global rescale by more than seed variance on at least the two larger shifts.

Why not just…?

Why not just rescale the input coordinates? This is the real threat. For a pure uniform scale change it may be all that is needed. It cannot, however, give different layers different effective scales, correct anisotropy, or reshape the spectrum. LiDAR scenes have range-dependent density and a vertical extent far smaller than the horizontal one. Deforming characters differ from rooms in both scale and local structure. The rescale baseline is tuned generously and reported first. If it matches our method on the scene and registration shifts, we will report that result and the contribution reduces to the diagnostic analysis.

Why not just use LoRA with a larger rank? The logit expression shows that LoRA never alters $\Omega$. It can only redistribute signal among existing bands. The rank-versus-$\kappa$ sweep tests this directly. If modest-rank LoRA matches spectrum adaptation at every $\kappa$, the structural argument fails in practice.

Why not just use GEM-style spatial adapters? They work, reaching 46.6 on ScanNet++. But they add convolution and cross-attention that cannot be folded into the weights, and they mix coordinate-dependent features into hidden states. GEM's own authors report trouble transferring to outdoor scenes. Our reimplementation on a shared backbone decides whether they help with geometric shift. We also test whether stacking them on top of spectrum adaptation adds anything.

Why not just retune the positional MLP, as Positional Prompt Tuning does? In a rotary model there is no input positional MLP to retune. Its closest analogue is our shared-coordinate-map ablation. PPT itself needed in-block adapters on top of the input module, which is consistent with position needing adjustment inside each layer.

Proposed Experiments

The registration track pretrains Lepard on 3DMatch and adapts it to 4DMatch and 4DLoMatch. The metric is non-rigid feature matching recall. We also check 3DLoMatch registration recall after adaptation to measure forgetting of the source domain; Lepard's in-domain figure there is 71.3%.

The segmentation track pretrains the rotary scene model on ScanNet v2 and adapts it to ScanNet++, S3DIS Area 5 and SemanticKITTI, measured in mIoU. The fractions of 1% and 10% of target data probe the low-data regime GEM highlights.

A controlled track rescales and anisotropically stretches ScanNet coordinates synthetically by $\kappa \in \{0.25, 1, 4, 16\}$. This produces the rank-versus-$\kappa$ curve and the attention-distance profiles.

The decisive checks are three. First, the gap between our method and the tuned global rescale must be positive on SemanticKITTI and 4DLoMatch. Second, the LoRA rank needed to match our method must grow with $\kappa$ in the controlled track. Third, the learned per-band scales must shift in the direction the source-versus-target attention profiles predict.

If the first check fails, the per-layer, per-band story is wrong. If the second fails, the structural argument against LoRA is irrelevant in practice. Either outcome is informative and would be reported.