Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis

Yaxuan Zhu, Ruiqi Gao, Siyuan Huang, Song-Chun Zhu, Yingnian Wu

TL;DR

This paper introduces a novel approach to represent camera poses and their movements in 3D space using learned neural representations. By modeling camera movements as matrix operations on pose vectors, their method achieves improved robustness to noise in novel view synthesis and better performance in camera pose regression compared to traditional methods.

What Problem Does This Paper Address?

Representing camera pose (position and orientation) is crucial for many 3D computer vision tasks like estimating camera location or generating new views of a scene. Traditional methods rely on manually designed representations like Euler angles or quaternions. While these have been effective, they might not be optimal for deep learning models, which often struggle with the inherent complexities and potential discontinuities of these representations. The authors aim to overcome this by learning "neural" representations that are more naturally integrated with deep networks and are more robust to real-world challenges like noisy sensor data.

Methods

The core of the proposed method lies in learning a vector representation for camera poses and a matrix representation for local camera movements.

  • Pose Representation: Instead of using fixed mathematical forms, each degree of freedom (position and orientation) is mapped to a high-dimensional vector. These vectors are learned from data.
  • Movement Representation: Local camera movements are modeled as transformations that operate on these pose vectors. Specifically, they use a matrix representation that, for infinitesimal movements, can be related to a matrix Lie algebra. This allows for a continuous and smooth representation of motion.
  • View Synthesis: The learned pose and scene representations are fed into a decoder network to generate a 2D image from a specific viewpoint. This generative task is used to train the pose and scene representations.
  • Pose Regression: The learned pose representation is also used as a target for a separate inference network, allowing the model to predict camera poses from images.

A key aspect is the use of a rotation loss that enforces consistency in how pose vectors transform, contributing to the learned representation's robustness and translation invariance.

Datasets

The authors evaluated their method on several datasets:

  • GQN Rooms: A synthetic dataset designed for novel view synthesis, featuring various indoor scenes.
  • ShapeNet v2: A dataset of 3D object models (cars) used to generate synthetic images for training and evaluation.
  • Gibson Environment: A dataset providing realistic indoor scenes for generating synthetic data.
  • 7 Scenes Dataset: A real-world dataset commonly used for camera pose estimation.

Key Insights and Ideas

  • Learned Representations Outperform Hand-Crafted Ones: The paper strongly suggests that learned, high-dimensional vector representations for camera pose are more effective and robust than traditional, manually designed representations like Euler angles or quaternions, especially when integrated with deep learning models.
  • Matrix Lie Algebra for Motion: Modeling camera movement using matrix Lie algebra provides a principled way to represent continuous transformations in the learned neural space, leading to better generalization.
  • Rotation Loss for Robustness: The introduction of a specific rotation loss is crucial for enforcing translation invariance and making the learned pose representations more resilient to noise.
  • Joint Learning of Pose and Scene: The framework jointly learns representations for both the camera pose and the 3D scene, allowing them to inform each other during the view synthesis process.

Key Results

  • Novel View Synthesis: The proposed method achieved comparable or better results than the state-of-the-art GQN model on various datasets, particularly in more complex scenes (Gibson rooms), where GQN struggled to capture scene relationships.
  • Robustness to Noise: The learned representation demonstrated significantly better robustness to noise added to the pose information compared to the GQN model, maintaining higher PSNR values.
  • Camera Pose Regression: The learned representation consistently outperformed traditional representations (Euler angles, quaternions, log quaternions, sinusoidal) in camera pose regression tasks across multiple datasets, achieving lower median errors. For instance, on the 7 Scenes dataset, their method achieved an average error of 0.21m and 7.92°, outperforming baselines.

Limitations and Future Work

  • Computational Cost: Training the model, especially for generative tasks, can be computationally intensive, requiring significant GPU time.
  • Future Work: The authors suggest exploring the integration of their learned representation with recent advancements like NeRF (Neural Radiance Fields) to potentially achieve more flexible and powerful camera pose representations. They also highlight the potential for further research into learning neural representations for camera poses and joint representations of poses and 3D scenes.

Summary updated

Abstract

How to effectively represent camera pose is an essential problem in 3D computer vision, especially in tasks such as camera pose regression and novel view synthesis. Traditionally, 3D position of the camera is represented by Cartesian coordinate and the orientation is represented by Euler angle or quaternions. These representations are manually designed, which may not be the most effective representation for downstream tasks. In this work, we propose an approach to learn neural representations of camera poses and 3D scenes, coupled with neural representations of local camera movements. Specifically, the camera pose and 3D scene are represented as vectors and the local camera movement is represented as a matrix operating on the vector of the camera pose. We demonstrate that the camera movement can further be parametrized by a matrix Lie algebra that underlies a rotation system in the neural space. The vector representations are then concatenated and generate the posed 2D image through a decoder network. The model is learned from only posed 2D images and corresponding camera poses, without access to depths or shapes. We conduct extensive experiments on synthetic and real datasets. The results show that compared with other camera pose representations, our learned representation is more robust to noise in novel view synthesis and more effective in camera pose regression.
BibTeX
@inproceedings{zhu2021learning,
  author = {Yaxuan Zhu and Ruiqi Gao and Siyuan Huang and Song-Chun Zhu and Yingnian Wu},
  title = {{Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=EPj5eDwroa4},
}
Figure