Aggregation of binary feature descriptors for compact scene model representation in large scale structure-from-motion applications

Jacek Komorowski, Tomasz Trzcinski

Abstract

In this paper we present an efficient method for aggregating binary feature descriptors to allow compact representation of 3D scene model in incremental structure-from-motion and SLAM applications. All feature descriptors linked with one 3D scene point or landmark are represented by a single low-dimensional real-valued vector called a prototype. The method allows significant reduction of memory required to store and process feature descriptors in large-scale structure-from-motion applications. An efficient approximate nearest neighbours search methods suited for real-valued descriptors, such as FLANN, can be used on the resulting prototypes to speed up matching processed frames.

Introduction

Large-scale Structure-from-Motion (SfM) and Simultaneous Localization and Mapping (SLAM) systems are fundamental to modern computer vision, enabling the construction of 3D models from thousands of images. These systems rely on detecting and tracking "landmarks" across frames. To facilitate loop closure and localization, systems typically store every feature descriptor (e.g., SIFT, FREAK) associated with a specific landmark.

As models scale to millions of points, memory consumption becomes a critical bottleneck, especially for mobile devices. While binary descriptors like FREAK are compact, they are difficult to aggregate. Simple bit-wise averaging (quantized mean) is not robust for incremental updates because discarding original descriptors loses the "voting history" needed for future refinements. This research proposes a method to represent multiple descriptors as a single, compact, and updatable "prototype" vector without sacrificing matching accuracy.

Proposed Method

The core of the proposed approach is a neural network that transforms high-dimensional binary descriptors into a low-dimensional, real-valued embedding space. This shift allows for more flexible aggregation and the use of advanced matching algorithms.

Comparison between the traditional multi-descriptor approach and the proposed prototype-based method.
  1. Embedding: A neural network $f(x)$ maps a 512-bit binary FREAK descriptor to a $k$-dimensional real-valued vector (e.g., $k=32$).
  2. Aggregation: For any given 3D landmark, the "prototype" is calculated as the arithmetic mean of the embeddings of all descriptors linked to it.
  3. Incremental Updates: Because the prototypes are real-valued, they can be updated via a simple weighted average whenever a new frame provides an additional descriptor for an existing landmark.
  4. Matching: New descriptors from incoming frames are embedded using the same network and matched against stored prototypes using Euclidean distance. This enables the use of efficient libraries like FLANN for fast approximate nearest neighbor (ANN) searches.

Training Strategy

The researchers explored two primary training paradigms: Triplet Networks and Prototypical Networks.

The triplet network architecture used to learn discriminative embeddings.

As shown in the triplet architecture, the network $f$ is trained to minimize the distance between an anchor and a positive sample ($x, x^+$) while maximizing the distance to a negative sample ($x^-$). Surprisingly, Triplet Networks consistently outperformed Prototypical Networks, despite the latter being specifically designed for prototype-based classification.

Architectural analysis revealed that shallow, three-layer fully connected networks were sufficient. The embedding quality remained robust regardless of whether the network maintained a constant width ("fat") or a decreasing width ("funnel") across layers.

Experimental Results

The method was evaluated using data from a Google Tango tablet, involving millions of FREAK descriptors. The results demonstrate a compelling trade-off between memory efficiency and precision:

Method NN Search Precision
Unaggregated descriptors (Baseline) 0.796
32-dimensional prototypes (Proposed) 0.750
16-dimensional prototypes (Proposed) 0.676
Quantized mean (512 bits) 0.671
Mean of PCA projection (16 dims) 0.581
Random sample (512 bits) 0.526

A 32-dimensional prototype (using 4-byte floats) consumes roughly the same space as two 512-bit FREAK descriptors. Given that landmarks often aggregate 3 to 15 descriptors, this method provides a massive reduction in the memory footprint of 3D models while maintaining precision close to the unaggregated baseline.

Conclusion and Future Work

This approach successfully compresses 3D scene models by aggregating binary descriptors into real-valued prototypes. While highly effective, the authors note an initial "initialization period" where raw descriptors must be stored for triangulation before switching to prototypes. Future work may involve incorporating keypoint metadata, such as scale and orientation, to further enhance the discriminative power of the prototypes and optimizing the computational overhead of the neural network inference during real-time processing.

Summary updated

Abstract

In this paper we present an efficient method for aggregating binary feature descriptors to allow compact representation of 3D scene model in incremental structure-from-motion and SLAM applications. All feature descriptors linked with one 3D scene point or landmark are represented by a single low-dimensional real-valued vector called a \emph{prototype}. The method allows significant reduction of memory required to store and process feature descriptors in large-scale structure-from-motion applications. An efficient approximate nearest neighbours search methods suited for real-valued descriptors, such as FLANN, can be used on the resulting prototypes to speed up matching processed frames.
BibTeX
@inproceedings{komorowski2018aggregation,
  author = {Jacek Komorowski and Tomasz Trzcinski},
  title = {{Aggregation of binary feature descriptors for compact scene model representation in large scale structure-from-motion applications}},
  booktitle = {ICCVG},
  year = {2018},
  url = {https://openreview.net/forum?id=cL76kPRTIPG},
}
Figure