TL;DR
This paper introduces a self-supervised method to train a deep network that identifies "distractors"—moving objects like buses and cars—in urban camera images. By using an "ephemerality mask" to ignore these regions, a single-camera system can maintain accurate, metric-scale motion estimation even when 90% of the view is blocked by a moving vehicle.
What Problem Does This Paper Address?
Monocular Visual Odometry (VO) is the process of estimating a vehicle's motion using only a single camera. In clean, static environments, this works well. However, urban centers are chaotic.
The primary challenge is the presence of distractors: independently moving objects like buses, trucks, and pedestrians. Standard outlier rejection schemes (like RANSAC) rely on the assumption that the majority of tracked features belong to the static "world." If a large bus pulls in front of your car and fills 80% of the frame, the VO system might mistakenly conclude that the bus is the static background and that your car is moving in the opposite direction of the bus. This leads to massive errors in ego-motion estimation.
Methods
The authors propose a pipeline that moves from high-end offline mapping to low-cost online execution:
- Self-Supervised Labeling: To avoid manual pixel-labeling, the authors used a survey vehicle equipped with LIDAR and stereo cameras to drive the same routes multiple times. By comparing these traversals, they calculated a structural entropy metric. If a 3D point appears consistently across many days, it’s "static" (buildings, road signs); if it appears rarely, it’s "ephemeral" (parked cars, traffic).
- Deep Learning: They trained a deep convolutional neural network (an encoder with two decoders) using only monocular images as input. The network learns to predict two things simultaneously:
- Metric Depth: Providing the absolute scale usually missing in monocular systems.
- Ephemerality Mask: A per-pixel probability map indicating which regions are likely to be moving distractors.
- Ephemerality-Aware VO: They integrated this mask into two types of VO:
- Sparse VO: The mask acts as a hard filter, preventing the system from picking features on moving objects.
- Dense VO: The mask acts as a weight in the photometric error function, effectively "muting" the influence of pixels identified as ephemeral.
Datasets
The study utilizes the Oxford RobotCar Dataset.
- Training: 80km of driving (8 traversals of a 10km route).
- Evaluation: Over 400km of driving across 42 different traversals. Crucially, the evaluation included "unmapped" locations not seen during training to ensure the model learned general features of "ephemerality" rather than just memorizing a specific map.
Key Insights and Ideas
- Entropy as Ground Truth: The "aha moment" is using repeated traversals to define what is static. This bypasses the need for semantic labels (e.g., "this is a bus"). If it moves, the entropy is low; if it stays, the entropy is high.
- Scale from Experience: By training the monocular network against LIDAR-derived depth, the system learns to estimate metric distance from a single lens, solving the classic "scale ambiguity" problem of monocular VO.
- Ignoring the Majority: Traditional systems fail when distractors dominate the frame. This approach proves that as long as the network can identify even a small sliver of static world (like a patch of pavement or a distant rooftop), it can maintain an accurate trajectory.
Key Results
The authors compared their "Ephemerality-Aware" versions against baseline sparse and dense VO:
- Drift Reduction: In the full 400km test, the ephemerality mask reduced translational drift. For example, Sparse VO drift dropped from 6.55% to 6.38%.
- Robustness to Traffic: The most dramatic results appeared in "distractor-heavy" scenes. In these scenarios, the velocity estimation error for Sparse VO was 0.0489 m/s with the mask, compared to 0.220 m/s without it—a 4.5x improvement in accuracy.
- Qualitative Success: The masks successfully identified a wide variety of dynamic objects, including strollers, cyclists, and pedestrians, even at varying distances and orientations.
Limitations and Future Work
- Training Requirements: The method requires an initial "survey" phase with expensive sensors (LIDAR/Stereo) to generate the self-supervised labels. It cannot learn "on the fly" in a completely new city without prior data.
- Semantic Integration: While the mask identifies that something is moving, it doesn't know what it is. The authors suggest that future work could combine this with semantic segmentation to better track specific object classes.
- Closed-Loop Control: The current paper focuses on estimation. Future work could explore how these robust estimates improve the actual path-planning and safety of autonomous vehicles in heavy traffic.
Summary updated