Kolibri: Color Finds the Place, Refinement Sets the Pose for RGB-D Localization in Colored Point Clouds

Leonel Aguilar, Bartu Tuncay, Christoph HoelscherarXiv 2026 · arXiv

TL;DR

Kolibri is a lightweight localization method that splits the task of finding a camera's pose into two steps: a tiny neural network uses color to find the approximate "place," and standard geometric refinement handles the precision. By leveraging the insight that many nearby initial guesses converge to the same correct pose, Kolibri achieves state-of-the-art accuracy while being 18x to 296x faster than existing methods, enabling real-time localization across entire buildings without pre-defined room labels.

What Problem Does This Paper Address?

Localization—the ability of a robot or agent to determine its exact position and orientation within a pre-mapped environment—is a fundamental challenge in robotics. While we have excellent tools for this, they often face a "speed vs. scale" trade-off.

Standard point cloud registration methods are highly accurate but computationally expensive, often taking seconds or even minutes per frame. To speed things up, many systems use a two-step "retrieve-then-align" workflow: first, an image retrieval system guesses which room the robot is in, and then a registration algorithm aligns the frame within that specific room. However, if the retrieval step picks the wrong room (which happens often in buildings with similar-looking hallways or offices), the localization fails entirely.

The authors of Kolibri set out to create a system fast enough to search an entire building's point cloud at once, removing the need for error-prone room labels.

Methods

The core philosophy of Kolibri is: "Color finds the place, refinement sets the pose."

  1. The Kolibri Network: Instead of a massive model, the authors use a tiny graph neural network (GNN) with only 55,712 parameters. It processes the environment as a graph of 10cm cubic "cells." Each cell stores information about its color, local shape, and its relationship to 16 neighboring cells.
  2. Candidate Proposing: The network matches cells from the current camera frame to cells in the global point cloud. It uses these matches to propose a few (up to 6) candidate poses.
  3. Geometric Refinement: Kolibri then hands these candidates over to a standard, non-learned algorithm: point-to-plane Iterative Closest Point (ICP).
The Kolibri pipeline: from raw RGB-D frames to a precise camera pose.

The key technical insight is the "refinement basin." The authors observed that if an initial pose guess is within a certain distance (about 7.7cm) of the truth, ICP will almost always converge to the correct final pose. This means the learned neural network doesn't need to be perfectly accurate; it just needs to land "inside the basin."

Figure 2. The network finds candidate places, and geometry refines their poses. Point-to-plane iterative closest point (ICP) aligns frame points to nearby surfaces. The dashed region illustrates the refinement basin. Its measured boundary is the initial position error at which half the starts succeed within 5 cm and 5° (Fig. 4a). A geometric fit score selects the final candidate.

Datasets

The researchers evaluated Kolibri on several standard benchmarks and one new real-world scenario:

  • ScanNet: A large-scale dataset of indoor scenes. They used specific subsets (ScanNet-10, ScanNet-96) to test generalization to unseen rooms.
  • 7-Scenes & 12-Scenes: Classic benchmarks for indoor relocalization.
  • Hospital Emergency Department: A new, challenging test case involving a real-world scan of a large hospital wing (49m x 42m) captured with a handheld laser scanner.

Key Insights and Ideas

  • Efficiency through Simplicity: By offloading the "heavy lifting" of precision to ICP, the neural network can remain incredibly small. This makes it feasible to run on low-power hardware or process massive environments quickly.
  • Color as a Primary Cue: Much like humans use the "gist" of a scene (colors and rough shapes) to recognize a room, Kolibri uses color to narrow down the search space rapidly.
  • Searching the Whole Building: Because Kolibri is so fast (82ms per frame), it can search a point cloud of 100 rooms simultaneously. This is much more robust than "retrieve-then-align" methods, which lose significant accuracy if the initial room retrieval is wrong.

Key Results

  • Speed: On an RTX 2080 Ti, Kolibri processes frames in 82ms, making it roughly 18x faster than BUFFER-X and 296x faster than GeDi.
  • Accuracy: Despite its small size, it matches the accuracy of much larger SOTA models, achieving 81.4% recall (within 5cm and 5°) on unseen ScanNet rooms.
  • Scale: In the hospital emergency department test, Kolibri localized 86.5% of frames in 384ms each. For comparison, the GeDi method required 91 minutes per frame to achieve similar results on the same hardware.
Figure 1. Kolibri searches a colored point cloud without a room label and matches the accuracy of BUFFER-X given the true room. (a) In the ten-room ScanNet-10 point cloud, image retrieval favors similar-looking views from other rooms (gray). Searching the full point cloud lets Kolibri localize this frame in its true room (orange). (b) With the room given, recall at 5 cm and 5° is plotted against median time per frame on an RTX 2080 Ti. Methods use their released settings and our refinement, with Kolibri in its standard Fast setting. Vertical bars are 95% intervals that resample whole walks of consecutive frames. Thin lines in a method’s color show its other settings (BUFFER-X, GeDi and RGB GeoTransformer).

Limitations and Future Work

The primary limitation of Kolibri is its dependence on color consistency.

  • Grayscale/Low Light: If the camera provides grayscale images or if the lighting is drastically different from the original scan, performance drops significantly. In these cases, purely geometric methods like RGB GeoTransformer still hold the lead.
  • Cross-Camera Calibration: The method is sensitive to differences in how different cameras respond to color.

Future work may involve creating more robust color-invariant features or developing a hierarchical approach to scale the system to even larger environments, like multi-story city blocks.

Summary updated

Abstract

An agent or robot in a scanned building must locate its camera in rooms that no model saw during training. Point cloud registration methods can align each RGB-D frame with the building's colored point cloud, accurately but at a high cost. We observe that geometric refinement brings nearby initial poses to the same final pose, so the learned component only needs to find the place. Kolibri does this with a graph network of 55,712 parameters that describes 10 cm cubes of points, called cells, by their color, local shape and neighbors. With the same refinement, it matches GeDi on 96 unseen ScanNet rooms (81.4% against 80.3% within 5 cm and 5°) and on a held-out test of 10 further rooms, in 82 ms per frame on an RTX 2080 Ti. The saved time allows for search without a room label. In one point cloud of 100 rooms, Kolibri loses 1.3 points against knowing the room, while retrieving the room first loses 56.0. In a hospital emergency department scene from a real scan, it localizes 86.5% of the frames rendered from the scan in 384 ms each, where GeDi needs 91 minutes per frame on cubes of the scan for a similar recall. The method needs colors that agree between the frame and the point cloud. On gray frames and across cameras with different color responses, point cloud registration methods keep the lead.
BibTeX
@misc{aguilar2026kolibri,
  author = {Leonel Aguilar and Bartu Tuncay and Christoph Hoelscher},
  title = {{Kolibri: Color Finds the Place, Refinement Sets the Pose for RGB-D Localization in Colored Point Clouds}},
  year = {2026},
  eprint = {2611.22829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22829},
}
Figure