Multimotion Visual Odometry (MVO): Simultaneous Estimation of Camera and Third-Party Motions

Kevin M. Judd, Jonathan D. Gammell, Paul Newman

Abstract

Estimating motion from images is a fundamental task in robotics. While traditional visual odometry (VO) focuses on camera motion in static scenes, and object tracking often requires prior models, estimating both simultaneously remains a challenge. This paper introduces Multimotion Visual Odometry (MVO), a framework that estimates the full $SE(3)$ trajectories of a moving camera and multiple independent objects without prior knowledge. Evaluated on a real-world dataset with ground truth, MVO demonstrates robust performance in complex dynamic environments.

Introduction

In robotic vision, Visual Odometry (VO) typically assumes a static world to estimate a robot's ego-motion. However, real-world environments like busy urban streets are inherently dynamic. Traditional VO systems often treat moving objects as outliers to be filtered out, which can lead to failure if dynamic objects dominate the field of view. Conversely, multiple object tracking (MOT) systems often rely on pre-trained detectors or assume known camera motion.

Multimotion estimation of a camera and four moving objects.

MVO addresses this "chicken-and-egg" problem: accurate segmentation requires motion information, while motion estimation requires segmented objects. By treating moving objects as valuable data rather than noise, MVO simultaneously estimates the 6-DOF trajectories of the camera and all rigid bodies in the scene.

Methodology

The MVO pipeline extends the standard stereo VO workflow by incorporating a robust multimotion fitting stage.

1. Tracklet Generation and Graph Construction

The system extracts sparse 3D features and matches them across frames to create "tracklets." A geometric neighborhood graph is then constructed where vertices represent tracklets. Edges connect tracklets that maintain spatial proximity over time, indicating they likely belong to the same rigid body.

2. Iterative Labeling via Energy Minimization

Unlike sequential RANSAC approaches that greedily extract motions, MVO employs an energy minimization framework called CORAL. This process involves:

  • Proposal: Generating motion hypotheses using RANSAC on small tracklet clusters.
  • Assignment: Labeling tracklets (e.g., "Camera," "Object N," or "Outlier") by balancing data fit against spatial smoothness.
  • Refinement: Dynamically merging similar motion models and splitting those that poorly explain the data.
Coordinate frames and transformations for multiple rigid bodies over time.

3. Batch Estimation

Following segmentation, a full-batch bundle adjustment is performed for each identified motion to produce smooth, high-accuracy $SE(3)$ trajectories.

Experimental Results

The authors validated MVO using a custom dataset captured with a Bumblebee XB3 stereo camera, with ground truth provided by a Vicon motion capture system for both the camera and four moving blocks.

Segmentation and Tracking Accuracy

In experiments involving five distinct motions (camera + 4 objects), MVO correctly identified the number of motions in 96.8% of frames, significantly outperforming sequential RANSAC (31.2%). The camera's ego-motion drift was approximately 3.24% of the path length, comparable to state-of-the-art static VO systems. Object tracking achieved translational errors between 0.27m and 0.99m, depending on the duration the objects remained within the camera's frustum.

Key Insights

  • Stability: Simultaneous estimation is far more stable than sequential methods, which often produce "ghost" motions from noise.
  • Generality: Being model-free, MVO tracks any rigid body based on geometry rather than visual appearance (e.g., car vs. box).
  • Smoothness: Graph-based regularization ensures clean segmentation even with sparse feature points.

Conclusion and Future Work

MVO provides a robust framework for understanding complex dynamic scenes without prior object models. Current limitations include a dependency on feature-rich surfaces and challenges with objects re-entering the field of view. Future work will focus on improving re-identification and optimizing the pipeline for real-time robotic applications.

Summary updated

Abstract

Estimating motion from images is a well-studied problem in computer vision and robotics. Previous work has developed techniques to estimate the motion of a moving camera in a largely static environment (e.g., visual odometry) and to segment or track motions in a dynamic scene using known camera motions (e.g., multiple object tracking). It is more challenging to estimate the unknown motion of the camera and the dynamic scene simultaneously. Most previous work requires a priori object models (e.g., tracking-by-detection), motion constraints (e.g., planar motion), or fails to estimate the full SE(3) motions of the scene (e.g., scene flow). While these approaches work well in specific application domains, they are not generalizable to unconstrained motions. This paper extends the traditional visual odometry (VO) pipeline to estimate the full SE(3) motion of both a stereo/RGB-D camera and the dynamic scene. This multimotion visual odometry (MVO) pipeline requires no a priori knowledge of the environment or the dynamic objects. Its performance is evaluated on a real-world dynamic dataset with ground truth for all motions from a motion capture system.
BibTeX
@inproceedings{judd2018multimotion,
  author = {Kevin M. Judd and Jonathan D. Gammell and Paul Newman},
  title = {{Multimotion Visual Odometry (MVO): Simultaneous Estimation of Camera and Third-Party Motions}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=r0JnX3becm},
}
Figure