Direct optimization of frame-to-frame rotation

Laurent Kneip, Simon LynenOpenReview 2013 · OpenReview

Abstract

This work makes use of a novel, recently proposed epipolar constraint for computing the relative pose between two calibrated images. By enforcing the coplanarity of epipolar plane normal vectors, it constrains the three degrees of freedom of the relative rotation between two camera views directly—independently of the translation. The present paper shows how the approach can be extended to $n$ points and translated into an efficient eigenvalue minimization over the three rotational degrees of freedom. Each iteration in the non-linear optimization has constant execution time, independently of the number of features. Two global optimization approaches are proposed: an efficient Levenberg-Marquardt scheme with randomized initial values and a globally optimal branch-and-bound algorithm. The algorithm shows state-of-the-art performance compared to essential-matrix based solutions and enables real-time visual odometry.

Introduction

Estimating the relative motion between two camera frames is a fundamental task in Structure from Motion (SfM) and Visual Odometry. Traditionally, this is solved using the Essential Matrix ($E$), which encapsulates both rotation ($R$) and translation ($t$). However, this coupling leads to several issues: solution multiplicity (four possible decompositions), structural degeneracy in planar scenes, and a complete breakdown in "zero-translation" scenarios where the camera only rotates.

Kneip and Lynen propose a paradigm shift by isolating rotation from translation. By focusing on the rotational component first, they create a solver that is more robust to common geometric degeneracies and significantly more efficient for large numbers of feature points.

Methodology

The core innovation is a novel epipolar constraint based on the coplanarity of epipolar plane normal vectors. In a standard two-view setup, for any observed point, the two bearing vectors and the translation vector must lie within the same epipolar plane. Consequently, the normal vectors of these planes across multiple points must also be coplanar.

Geometric setup showing bearing vectors, epipolar planes, and normal vectors.

The authors stack these normal vectors into a matrix $N$. The objective is to find the rotation $R$ that minimizes the smallest eigenvalue ($\lambda_{min}$) of the matrix $M = NN^T$. This effectively finds the rotation that makes the set of normal vectors as "flat" as possible in one direction.

Computational Efficiency

A major technical advantage of this formulation is that the matrix $M$ can be pre-calculated in $O(n)$ time. Once $M$ is computed, the subsequent non-linear optimization iterations have a constant execution time $O(1)$, regardless of whether there are 10 or 10,000 points. This makes the approach exceptionally well-suited for high-rate real-time applications.

Experimental Results

The proposed "eig" algorithm was evaluated against standard solvers, including Nistér’s 5-point and Stewénius’ 6-point algorithms, using both synthetic datasets and real-world video sequences with Vicon ground truth.

Comparison of translation direction error across different solvers as a function of image noise.

As shown in the performance plots, the iterative "eig" solver consistently achieves lower rotation and translation errors in the presence of noise. In real-world tests, the method maintained rotation accuracy better than 1° per frame pair. Furthermore, the median execution time was recorded at approximately 83 microseconds, demonstrating its suitability for high-speed robotics.

Conclusion

By decoupling rotation from translation, this method provides a stable and efficient alternative to traditional essential-matrix-based pose estimation. It remains robust in pure-rotation scenarios and offers a constant-time optimization loop. While the current implementation relies on RANSAC for outlier rejection, the underlying eigenvalue minimization provides a powerful foundation for future real-time visual odometry systems.

Summary updated

Abstract

This work makes use of a novel, recently proposed epipolar constraint for computing the relative pose between two calibrated images. By enforcing the coplanarity of epipolar plane normal vectors, it constrains the three degrees of freedom of the relative rotation between two camera views directly—independently of the translation. The present paper shows how the approach can be extended to n points, and translated into an efficient eigenvalue minimization over the three rotational degrees of freedom. Each iteration in the non-linear optimization has constant execution time, independently of the number of features. Two global optimization approaches are proposed. The first one consists of an efficient Levenberg-Marquardt scheme with randomized initial value, which already leads to stable and accurate results. The second scheme consists of a globally optimal branch-and-bound algorithm based on a bound on the eigenvalue variation derived from symmetric eigenvalue-perturbation theory. Analysis of the cost function reveals insights into the nature of a specific relative pose problem, and outlines the complexity under different conditions. The algorithm shows state-of-the-art performance w.r.t. essential-matrix based solutions, and a frameto-frame application to a video sequence immediately leads to an alternative, real-time visual odometry solution.
BibTeX
@misc{kneip2013direct,
  author = {Laurent Kneip and Simon Lynen},
  title = {{Direct optimization of frame-to-frame rotation}},
  year = {2013},
  url = {https://openreview.net/forum?id=SkNPmZGOZS},
  howpublished = {OpenReview},
}
Figure