A person who says "pick me up by the yellow building next to the bus stop" is giving two kinds of evidence. "Next to the bus stop" is about layout, and layout survives night, snow and rain. "Yellow" is about appearance. In a 3D map built from laser scans (a point cloud, i.e. a set of measured 3D points), appearance only exists because camera colors captured on one particular day were painted onto the geometry. Current systems that localize a sentence in such a map let every word attend to every feature channel of every map object. The strongest of them, Text2Loc++, reports two weaknesses. It loses about 10% recall when descriptions become long and conversational, and it degrades significantly when the map has no color. We argue these are one failure: the model has no way to decide, word by word, whether a word should be matched against geometry, against color, or against nothing.
We propose a cross-attention layer in which text tokens query map objects. Cross-attention here means a layer where features from one modality, the words, decide how much to read from features of another, the map objects. In our layer, each map object's feature is split into a geometry part and an appearance part. Each word carries a learned gate that splits its attention between the two parts and an explicit "null" slot that lets filler words attend to nothing. The appearance branch is additionally scaled by an estimate of how trustworthy the map's colors are. The claim is that this single mechanism recovers most of the complex-prompt drop and makes accuracy degrade gracefully, rather than sharply, as map colors become stale, shifted or absent. In that last case the model falls back to a geometry-only localizer at no extra cost.
Where this sits in prior work
The language-to-point-cloud line motivates the problem and supplies the backbone. Text2Pos introduced the KITTI360Pose benchmark and a two-step pipeline: retrieve a candidate map cell, then match textual hints to segmented objects. RET injected relative object displacements into attention to separate repeated landmarks. Text2Loc replaced hard matching with a cascaded cross-attention transformer in which text queries point features, doubling RET's top-1 recall. Text2Loc++ added multi-city training, three prompt-complexity tiers, and masked instance training, which hides unmentioned objects so the model learns to ignore map clutter. We reuse Text2Loc++ wholesale and change only the fine cross-attention and the object encoder.
Masked instance training handles clutter on the map side. Our null slot and gate handle noise on the text side and appearance unreliability. Neither has been addressed.
The appearance-variance literature constrains the design. PointNetVLAD showed that point geometry survives day-to-night changes that break image retrieval, which is why geometry must stay the anchor. MDFL splits image descriptors into condition-invariant and condition-related parts by penalizing mutual information between them. We borrow that split for object features, but replace its adversarial training with a simpler decorrelation penalty, since the paper itself flags adversarial instability. CPO weights map regions by their consistency with the query and filters changed areas. Our reliability scale plays the same role, but it acts on feature channels rather than regions, and it is driven by language.
The 2D–3D gap enters through the observation, made concrete by RayI2P and the visual-versus-geometric-interest mismatch noted in the file, that photometric and geometric signals rarely coincide. Point-cloud color is a 2D quantity projected onto 3D, so it inherits every condition sensitivity of 2D imagery.
The attention mechanism itself comes from Gated Cross Attention, which mixes cross-modal context with original features through a learned gate to stop one stream from washing out the other. GCA uses a gate per feature stream. Ours is per word and per evidence type, and adds a null option. We test whether that granularity matters.
Core mechanism
For a word token $t$ and map object $j$, let $k^{g}_j, v^{g}_j$ be keys and values from the geometry part of the object feature (category, shape, position offset) and $k^{a}_j, v^{a}_j$ from the appearance part (color statistics). The token's output is
$$ o_t = \alpha^{g}_t \sum_j \mathrm{softmax}_j(q_t k^{g}_j)\, v^{g}_j \;+\; \alpha^{a}_t\, r \sum_j \mathrm{softmax}_j(q_t k^{a}_j)\, v^{a}_j \;+\; \alpha^{0}_t\, v^{0}, $$
where $q_t$ is the token's query vector, and $(\alpha^{g}_t, \alpha^{a}_t, \alpha^{0}_t)$ is a three-way softmax predicted from the token in its sentence context. These weights express how much the word relies on geometry, appearance, or nothing. $v^{0}$ is a learned null value. $r \in [0,1]$ is a map-level appearance reliability score, predicted from simple color statistics of the submap (saturation, brightness histogram, fraction of uncolored points).
Training adds the decorrelation penalty between geometry and appearance parts. It also uses appearance corruption: at random, map colors are darkened, hue-shifted, desaturated or removed. The model therefore learns to lower $r$, and to move weight from $\alpha^{a}$ to $\alpha^{g}$, when color evidence is bad. Without that training signal the gate would have no reason to fall back.
Research Program
We first build the corruption suite and the evaluation protocol. The suite applies controlled, test-time map appearance shifts to KITTI360Pose and to the Text2Loc++ multi-city data. Severity is graded, and the extreme is fully uncolored geometry. Crossing this suite with the Simple, Moderate and Complex prompt tiers gives a two-axis robustness grid. We then implement the layer inside the Text2Loc++ fine stage and train with identical data and schedule.
Finally, we analyze the learned gates. If the mechanism works for the stated reason, color and material words should carry high $\alpha^{a}$ on clean maps, and that weight should drop as severity rises. Spatial prepositions and object categories should load on $\alpha^{g}$. Conversational filler in Complex prompts should load on $\alpha^{0}$. We check this against word categories derivable from the template grammar of KITTI360Pose, which tells us exactly which tokens are colors, categories and relations.
Baselines
The naive reference point is Text2Loc++ trained and tested as published, evaluated on our corruption grid. On clean KITTI360Pose, Text2Loc++ reports 44% top-1 recall within 5 m for fine localization and 35.3% top-1 recall for global place recognition. Text2Loc reports 0.33 top-1 recall within 5 m. For uncolored maps, the paper states only that performance degrades significantly; no reported result found on Lacuna for Text2Loc++ on geometry-only maps. No per-split numbers were found for the multi-city benchmark either. We will therefore rerun Text2Loc++ ourselves on every cell of the grid with the released configuration, and that rerun is the reference.
The strongest competitor is Text2Loc++ retrained with the same appearance-corruption augmentation we use. If augmentation alone closes the gap, our layer is unnecessary. A second strong competitor is a geometry-only Text2Loc++ trained without color, which is the natural upper bound under full color loss. Our method should match it there and beat it on clean maps. All models share the same data, the same backbone (frozen T5 plus hierarchical transformer, PointNet++ objects), the same epochs, and the same hyperparameter search budget. Results are averaged over three seeds.
Ablations isolate the ingredients. We remove the null slot, and separately replace the per-word gate with a single GCA-style gate per stream, which tests whether word-level granularity matters. We drop the reliability scale $r$, and we drop the decorrelation penalty.
We count it as beating the baselines if three things hold. On clean maps we must stay within one point of the 44% top-1 recall at 5 m. On the Complex tier we must close at least half of the documented roughly 10% Simple-to-Complex gap. Under severe and full color loss we must stay within two points of the geometry-only model. And the augmented Text2Loc++ must fail to achieve all three at once.
Why not just…?
Why not just train Text2Loc++ with color augmentation? This is the real threat. Augmentation may teach the existing cross-attention to discount color on its own. Two considerations cut against it. A model trained to distrust color everywhere should lose accuracy on clean maps, where "yellow building" is discriminative. And augmentation does nothing about Complex-prompt filler. The augmented baseline is in the main comparison for exactly this reason. If it matches us on both axes, we will report that the mechanism is unnecessary.
Why not just rewrite complex prompts into simple ones with a language model? Text2Loc++'s own tiers show that Simple prompts localize better, so rewriting could recover much of the gap. However, a rewriter cannot see the map, so it cannot know which colors are trustworthy. Ungrounded text planning tends to produce descriptions that do not map onto physical layouts, as StruVis and CoCo argue in generation. Rewriting also adds language-model latency on top of a 2.27 ms localizer. We run rewritten-Complex prompts through the unmodified Text2Loc++ as an extra baseline on the Complex tier, and we also combine rewriting with our layer to check whether the two are complementary.
Why not just drop color entirely? PointNetVLAD shows that geometry alone survives illumination change. This is the right answer when color is useless. However, the Text2Loc++ degradation without color shows that color carries real signal when it is valid, and Text2Pos found that extra non-geometric cues such as street names add up to 20 points. The geometry-only model is our lower-envelope baseline. The claim is that our model tracks the better of "color" and "no color" at every severity, which neither fixed choice can do.
Proposed Experiments
The main experiment fills the two-axis grid of five color-severity levels by three prompt tiers. It uses KITTI360Pose for fine localization (top-1, top-5 and top-10 recall at 5, 10 and 15 m) and the Text2Loc++ multi-city benchmark for cross-city transfer. Latency is reported per query against the 2.27 ms reference.
The decisive check is the shape of the curves. As severity rises, our method's recall should bend smoothly from the colored-model level to the geometry-only level. The augmented baseline should either sit lower on clean maps or drop below geometry-only at high severity.
A second decisive check is mechanistic. The appearance weight on template color words should fall monotonically with severity. The null weight on Complex-tier filler tokens should exceed that on Simple-tier tokens. If the gates show no such structure while accuracy still improves, the gain came from extra capacity rather than routing, and we would say so.
References
- Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language
- Text2Pos: Text-to-Point-Cloud Cross-Modal Localization
- Text to Point Cloud Localization with Relation-Enhanced Transformer
- Text2Loc: 3D Point Cloud Localization from Natural Language
- PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition
- A Multi-Domain Feature Learning Method for Visual Place Recognition
- CPO: Change Robust Panorama to Point Cloud Localization
- RayI2P: Learning Rays for Image-to-Point Cloud Registration
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Vision
- CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generation