Sparse-Dense Motion Modelling and Tracking for Manipulation without Prior Object Models

Christian Rauch, Ran Long, Vladimir Ivan, Sethu Vijayakumar

Introduction

In robotic manipulation, the ability to interact with novel objects in unstructured environments remains a significant hurdle. Traditional systems often rely on "prior models"—pre-existing CAD files or extensive training data—to recognize and manipulate objects. However, in "in the wild" scenarios, a robot must handle thousands of unforeseen items. This paper introduces MultiMotionFusion (MMF), a hybrid framework that enables robots to track and grasp completely unknown objects by combining robust sparse keypoint matching with precise dense geometric modeling.

The Sparse-Dense Hybrid Approach

Existing model-free tracking methods typically fall into two categories: dense methods and sparse methods.

  1. Dense methods (such as Iterative Closest Point, or ICP) provide the geometric detail necessary for precise grasping but are prone to failure during rapid camera or object movement, as they often get "lost" between frames.
  2. Sparse methods (using keypoints) are remarkably resilient to fast motion but lack the surface information required for planning finger placement.

MMF bridges this gap by using sparse features to initialize dense tracking. By using sparse keypoints to "jump" to the correct neighborhood first, the dense refinement becomes remarkably resilient to high-velocity motion.

System Architecture

The MMF pipeline operates in four distinct phases to ensure robust, model-free tracking:

  • Estimation: The system extracts sparse features using SuperPoint. It uses these to calculate an initial "rough" pose via RANSAC. This estimate initializes a dense ICP refinement, preventing the tracker from failing during fast rotations.
  • Segmentation: To distinguish objects from the background, the system uses a Conditional Random Field (CRF). It combines sparse reprojection errors with optical flow to label pixels that move together as a single rigid entity.
  • Modelling: Once segmented, 3D points are fused into a persistent model. The robot builds a real-time database of objects it has encountered.
  • Redetection: If an object leaves the frame and returns, the system compares current sparse keypoints against its database of "lost" models. If a match is found, it resumes tracking the existing model, allowing for long-term autonomy and memory.

Experimental Evaluation

The researchers evaluated MMF using a Kawada Nextage dual-arm robot equipped with an Azure Kinect RGB-D sensor. Ground truth was established using a Vicon motion capture system to provide millimeter-accurate trajectories. The benchmarks included high-speed 180-degree torso rotations and manipulation tasks involving common grocery items (e.g., Jaffa cakes, porridge oats) on a moving conveyor belt.

Key Results

  • Robustness to Speed: In high-speed rotation tests where standard dense trackers like ElasticFusion or Co-Fusion failed completely, MMF maintained tracking with an Absolute Trajectory Error (ATE) of only 2.25 cm.
  • Grasping Accuracy: The system achieved a translational ATE of approximately 1.0 to 1.3 cm when tracking objects on a moving conveyor, which is precise enough for successful robotic grasping.
  • Efficiency: The entire pipeline runs at approximately 8 Hz (126ms per frame), making it fast enough for real-time interaction on modern hardware (tested on an Intel i9 with an RTX 2080 Super).

Discussion and Future Work

The core insight of this research is that motion itself can serve as a primary cue for object segmentation and tracking, bypassing the need for prior shape models. By storing a history of sparse descriptors, the robot avoids "goldfish memory," allowing it to reuse grasp information for objects it hasn't seen for several minutes.

However, limitations remain. The current sparse descriptors are only robust to about 45 degrees of in-plane rotation, and the CRF segmentation can occasionally "bleed" into the background if optical flow is noisy. Future directions include tighter integration between keypoint correspondence and optical flow tasks to improve pixel-level association over longer distances.

Summary updated

BibTeX
@article{rauch2022sparse,
  author = {Christian Rauch and Ran Long and Vladimir Ivan and Sethu Vijayakumar},
  title = {{Sparse-Dense Motion Modelling and Tracking for Manipulation without Prior Object Models}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  doi = {10.1109/lra.2022.3200177},
}
Figure