This paper, presented at IROS 2018, introduces a framework for Multimotion Visual Odometry (MVO). While traditional visual odometry (VO) treats moving objects as "noise" to be ignored, MVO treats them as valuable data, estimating the full 3D trajectories of both the camera and every independent moving object in the scene simultaneously.
TL;DR
MVO is a model-free system that simultaneously estimates the 6-DOF trajectories of a moving camera and all dynamic objects in a scene. By combining graph-based segmentation with iterative energy minimization, it avoids the need for pre-trained object detectors or simplified motion constraints (like assuming objects only move on a flat floor).
What Problem Does This Paper Address?
In the world of robotic vision, "Visual Odometry" is the standard way to figure out where a robot is going by looking at how static landmarks move in its field of view. However, most VO systems break down in dynamic environments (like a busy street) because they assume the world is static.
Existing solutions usually fall into two camps:
- Filtering: They try to detect and ignore moving objects (e.g., using a car detector). This requires knowing what you're looking for ahead of time.
- Flow-based methods: They calculate the velocity of every pixel (Scene Flow). While detailed, these methods often fail to group pixels into "objects" or provide full 3D rotation and translation ($SE(3)$) trajectories.
The authors tackle the "chicken-and-egg" problem of multimotion estimation: you need to segment the scene into objects to calculate their motion, but you need to know their motion to segment them accurately.
Methods
The MVO pipeline extends the traditional stereo VO pipeline with a "Robust Multimotion Fitting" stage.
- Tracklet Generation: The system detects and matches sparse 3D features across frames to create "tracklets"—short histories of a point's movement in 3D space.
- Graph Construction: It builds a geometric neighborhood graph where each vertex is a tracklet. Edges connect tracklets that stay close to each other over time, suggesting they belong to the same rigid body.
- Iterative Labeling (The Core Innovation): Instead of using a simple RANSAC approach (which is "greedy" and prone to noise), MVO uses an energy minimization framework called CORAL.
- Propose: It suggests new motion labels by running RANSAC on small groups of tracklets.
- Assign: It assigns every tracklet a label (e.g., "Camera," "Object 1," or "Outlier") by balancing how well the motion model fits the data against how "smooth" the segmentation is across the graph.
- Merge/Split: It dynamically merges labels that describe the same motion and splits labels that are failing to explain their data.
- Batch Estimation: Once the scene is segmented, it performs a full-batch bundle adjustment for each motion to produce smooth, high-accuracy trajectories.
Datasets
The authors evaluated MVO on a newly collected real-world dataset.
- Hardware: A Bumblebee XB3 stereo camera.
- Ground Truth: A Vicon motion capture system provided millimeter-accurate trajectories for the camera and four independent moving blocks.
- Significance: Most existing datasets only provide ground truth for the camera or use synthetic data. This dataset provides ground truth for every moving part of a complex, real-world scene.
Key Insights and Ideas
- Moving Beyond "Greedy" RANSAC: The paper demonstrates that "Sequential RANSAC" (finding the biggest motion, removing it, and repeating) is highly inconsistent. It often overfits to noise, finding "ghost" motions. MVO’s simultaneous estimation is much more stable.
- Model-Free Flexibility: Because MVO relies on rigid-body geometry rather than appearance, it doesn't care if it's looking at a car, a pedestrian, or a random cardboard box. If it moves rigidly, MVO can track it.
- The Power of Smoothness: By using a graph-based approach, the system "encourages" neighboring points to move together. This helps maintain a clean segmentation even when feature points are sparse.
Key Results
- Segmentation Accuracy: In a five-motion experiment (camera + 4 blocks), MVO correctly identified the number of motions 96.8% of the time, whereas Sequential RANSAC only succeeded 31.2% of the time.
- Camera Egomotion: The camera's trajectory drift was roughly 3.24% of the total path length, which is competitive with state-of-the-art VO systems that ignore dynamic objects entirely.
- Object Tracking: The system successfully estimated the full $SE(3)$ trajectories for all four blocks, with translational errors ranging from 0.27m to 0.99m depending on how long the object stayed in the camera's view.
Limitations and Future Work
- Feature Dependency: Like all sparse methods, MVO fails if an object is featureless (e.g., a smooth white van) or too small to provide enough tracklets.
- Field of View: Gaps in trajectories occurred when objects partially exited the camera's "frustum." The system currently struggles to re-identify an object once it leaves and re-enters the frame.
- Centroid Bias: Since the system only sees the surface of an object, the calculated "center" of the object's motion is just the average of the visible points. This can shift as the object rotates and different parts become visible.
- Real-time Performance: While the paper focuses on accuracy, the authors note that implementing this pipeline for real-time robotic use is a key next step.
Summary updated