TL;DR
This paper introduces I-MOVE, the first publicly available RGB-D/stereo dataset specifically designed for estimating the velocities of independently moving objects. The authors highlight the significant gap in existing datasets for this task and present I-MOVE as a crucial resource for developing and evaluating more robust velocity estimation models.
What Problem Does This Paper Address?
Estimating the velocity of independently moving objects from video data is a surprisingly challenging and underexplored problem in computer vision. While motion parameter estimation is vital for applications like robotic navigation, collision detection, and sports analytics, current methods struggle, especially in unconstrained real-world scenarios. Existing datasets often lack the necessary ground truth (like precise velocity measurements), diverse object types, or varied environments to train generalizable models. Furthermore, many approaches rely on specialized sensors (like IMUs) or egocentric views, which are not always feasible. This paper aims to bridge this gap by providing a comprehensive dataset and a baseline for this critical task.
Methods
The paper introduces the I-MOVE dataset, which is the primary contribution. The dataset is designed to facilitate the development of generalizable velocity estimation models. Key features include:
- RGB-D and Stereo Data: The dataset captures synchronized RGB-D and stereo video streams.
- Multiple Cameras: Data is collected using seven different cameras (Stereo RGB, RGB-D (IR), and Monochromatic IR) with varying specifications (resolution, FOV, FPS) to encourage research into camera-agnostic or camera-aware models.
- Ground Truth Velocity: Instantaneous velocity ground truth is provided using three Doppler radars, synchronized with the camera data.
- Physics-Based Validation: To ensure the accuracy of the radar ground truth, the authors designed specific experiments (free fall, rolling down a ramp, pendulum swings) where velocity could be calculated using physics laws.
- Diverse Scenes and Objects: The dataset includes a variety of indoor and outdoor scenes with multiple moving objects (pedestrians, cyclists, rolling objects) and motions.
For their baseline, the authors employ a multi-stage approach:
- Object Detection: A mild Gaussian blur is applied to color frames, followed by a background subtractor (KNN) to identify motion and initialize a bounding box for the object of interest.
- Object Tracking: The detected bounding box is used to initialize a Discriminative Correlation Filter (CSRT) tracker, which tracks the object across frames.
- 3D Projection: The 2D bounding box from the tracker is projected into 3D space using the camera's API to obtain the object's 3D coordinates.
- Kalman Filter: A Kalman filter, assuming constant acceleration, is used to estimate the object's velocity and its magnitude from the sequence of 3D coordinates.
Datasets
The paper introduces the I-MOVE dataset as its primary contribution. This dataset is designed to address the limitations of existing benchmarks for velocity estimation of independently moving objects. It features:
- Synchronized RGB-D and Stereo Videos: Captured from seven different cameras.
- Ground Truth: Instantaneous velocity from three Doppler radars, validated using physics-based calculations for free fall, rolling objects, and pendulums.
- Diverse Scenarios: Indoor and outdoor scenes, various objects (pedestrians, bikes, rolling objects), and different motion patterns.
The paper also references other datasets in related fields, such as:
- Multi-Object Tracking (MOT) Benchmark: Primarily for tracking, not direct velocity estimation with metric ground truth.
- Human Activity Recognition datasets (e.g., Human Activity Recognition dataset, UTD-MHAD, HumanEva): These datasets often rely on IMU data or have limited motion variety and lack precise velocity ground truth.
- RGB-D SLAM datasets (e.g., A Benchmark for the Evaluation of RGB-D SLAM Systems): These provide sensor trajectory ground truth but not object velocity.
- DIML RGB-D dataset: Contains RGB-D data but lacks velocity ground truth.
Key Insights and Ideas
- The critical need for a dedicated RGB-D/stereo dataset for independently moving object velocity estimation: Existing datasets are insufficient, forcing researchers to rely on simulations or incomplete real-world data.
- The importance of diverse camera setups: Using multiple cameras with different characteristics (FOV, resolution, depth sensing technology) is crucial for developing models that generalize well to various sensors and scenarios.
- The challenge of obtaining accurate ground truth velocity: The paper highlights the complexity of synchronizing radar data with camera frames and the necessity of physics-based validation to ensure the reliability of radar measurements.
- The limitations of current baseline methods: Even for relatively simple motions like rolling objects, current state-of-the-art approaches struggle, indicating a significant research gap.
- The potential for stereo-radar fusion: The authors propose future work on novel stereo-radar approaches, suggesting that combining vision and radar data could lead to more robust velocity estimation.
Key Results
The paper presents baseline results using the I-MOVE dataset, summarized in Table 3. The Mean Absolute Difference (MAD) in meters per second is used as the evaluation metric.
- Indoor/Outdoor Ramp Scenes: For simpler scenarios like objects rolling down a ramp (both indoor and outdoor), the RealSense 415 camera achieved a relatively good MAD of 0.283 m/s (compared to radar) and 0.275 m/s (compared to physics). The ZED camera also performed reasonably well with MADs of 1.090 m/s and 0.108 m/s respectively.
- Pendulum and Dropped Object Scenes: Results for more complex motions like pendulums and dropped objects were substantially worse. For instance, the RealSense 415 struggled significantly with dropped objects, showing high MADs (e.g., 2.083 m/s compared to radar).
- Multiple Moving Objects: The paper notes that baseline results for multiple moving objects are "even worse," underscoring the difficulty of this specific scenario.
Overall, the results demonstrate that while some progress has been made for simpler cases, the problem of independently moving object velocity estimation is far from solved, especially for dynamic and complex scenarios.
Limitations and Future Work
Limitations:
- Baseline Performance: The current baseline method, while functional, exhibits significant limitations, particularly for fast-moving or complexly moving objects. The object detection method can miss initial movements, and the tracker can struggle with occlusions or rapid changes in appearance.
- Radar Limitations: While radars provide ground truth, they have limitations, such as a wide beam that might not precisely isolate small objects and a limited number of velocity estimates. Consolidating radar data for multiple moving objects is noted as tedious and left for future work.
- Physics-Based Validation: While valuable, physics-based calculations for ground truth have assumptions (e.g., zero friction) that might not perfectly reflect real-world conditions.
Future Work:
- Novel Stereo-Radar Approaches: Develop methods that fuse stereo vision data with radar measurements to extract more accurate 3D velocity vectors.
- Dynamic Camera Data: Extend the dataset to include sequences captured with moving cameras, which is a common scenario in real-world applications.
- Improved Baseline: Further refine the object detection and tracking components of the baseline to handle more challenging scenarios.
- Dataset Expansion: The authors plan to continue updating and expanding the I-MOVE dataset with new scenes, objects, and potentially more advanced ground truth collection methods.
Summary updated