DistillPose: Lightweight Camera Localization Using Auxiliary Learning

Yehya Abouelnaga, Mai Bui, Slobodan Ilic

Abstract

We propose a lightweight retrieval-based pipeline to predict 6DOF camera poses from RGB images. Our pipeline uses a convolutional network (CNN) to encode a query image as a feature vector. A nearest neighbor lookup finds the pose-wise nearest database image. A siamese convolutional neural network regresses the relative pose from the nearest neighboring database image to the query image. The relative pose is then applied to the nearest neighboring absolute pose to obtain the query image's final absolute pose prediction. Our model is a distilled version of NN-Net that reduces its parameters by 98.87%, information retrieval feature vector size by 87.5%, and inference time by 89.18% without a significant decrease in localization accuracy.

Introduction

Determining a camera's precise 6-degree-of-freedom (6DOF) pose from a single RGB image is a foundational task for autonomous systems and augmented reality. While traditional methods rely on matching 2D points to 3D maps, they are often computationally prohibitive for edge devices. Recent deep learning approaches have introduced retrieval-based methods like NN-Net, which find a similar image in a database and calculate a relative offset. However, these models remain bulky. DistillPose addresses this efficiency gap, providing a "distilled" model optimized for real-time performance on resource-constrained hardware.

Methodology

The DistillPose pipeline operates in two primary stages: Information Retrieval and Relative Pose Regression. First, a query image is encoded into a compact feature vector to find the closest match in a training database. Once retrieved, a siamese network calculates the relative pose difference between the query and the database image.

Overview of the DistillPose retrieval-based pipeline.

Layerwise Training and Auxiliary Learning

A key innovation in this work is the use of layerwise training. Rather than relying solely on the final layer of a ResNet34 backbone, the authors attached pose regression heads to each of the four residual blocks. During training, the model is further enhanced through auxiliary geometric tasks, such as predicting frustum overlap and angular distance, which forces the network to learn more robust geometric features.

Model Distillation

The final "DistillPose" model is a significantly pruned version of the full architecture. By identifying that earlier layers (L1 and L2) often capture more relevant geometric information for localization than deeper semantic layers, the authors were able to use only the first residual block for the final lightweight version.

Detailed architecture showing layerwise pose regression heads and the compact DistillPose (Short Version).

Experimental Results

The model was evaluated on the Seven Scenes dataset, a standard benchmark for indoor relocalization. Results demonstrate that DistillPose achieves performance comparable to state-of-the-art models while being a fraction of the size:

  • Efficiency Gains: Model size was reduced by 98.87% (from 22.3M to 248K parameters), and inference time dropped by 89.18% (from 37ms to 4ms).
  • Storage: The retrieval feature vector size was reduced by 87.5%, significantly lowering database storage requirements.
  • Accuracy: The best variant (L1 with auxiliary losses) achieved a median error of 0.23m and 9.85°, staying remarkably close to the much larger NN-Net (0.21m and 9.30°).

Discussion and Conclusion

DistillPose demonstrates that for camera localization, deeper architectures are not always superior. Mid-level geometric features from earlier CNN layers are highly effective for relative pose regression. By combining layerwise training with geometric auxiliary losses, the authors created a model capable of real-time 6DOF estimation on the edge. Future work will explore scaling this approach to larger outdoor environments and further optimizing backbones specifically for geometric tasks.

Summary updated

Abstract

We propose a lightweight retrieval-based pipeline to predict 6DOF camera poses from RGB images. Our pipeline uses a convolutional neural network (CNN) to encode a query image as a feature vector. A nearest neighbor lookup finds the pose-wise nearest database image. A siamese convolutional neural network regresses the relative pose from the nearest neighboring database image to the query image. The relative pose is then applied to the nearest neighboring absolute pose to obtain the query image's final absolute pose prediction. Our model is a distilled version of NN-Net that reduces its parameters by 98.87%, information retrieval feature vector size by 87.5%, and inference time by 89.18% without a significant decrease in localization accuracy.
BibTeX
@inproceedings{abouelnaga2021distillpose,
  author = {Yehya Abouelnaga and Mai Bui and Slobodan Ilic},
  title = {{DistillPose: Lightweight Camera Localization Using Auxiliary Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=DqLk0HQ7yr},
}
Figure