MSL-RAPTOR: A 6DoF Relative Pose Tracker for Onboard Robotic Perception

Benjamin Ramtoula, Adam Caccavale, Giovanni Beltrame, Mac Schwager

TL;DR

MSL-RAPTOR is a lightweight, monocular 6DoF (six Degree of Freedom) tracking algorithm designed specifically for onboard robotic perception. By combining deep-learning-based 2D tracking with a class-specific Unscented Kalman Filter (UKF), it achieves state-of-the-art accuracy and runs three times faster than comparable methods on embedded hardware like the NVIDIA Jetson TX2.

What Problem Does This Paper Address?

For a robot—particularly a fast-moving drone—to interact with its world, it needs to know exactly where other objects are (3D position) and how they are oriented (3D rotation). This is known as 6DoF relative pose tracking.

While high-end computer vision models can solve this, they usually require massive GPUs or depth-sensing cameras (RGB-D). Depth cameras are often impractical for outdoor use or long-range tracking, and heavy computation drains the batteries of small robots. Existing monocular (single-camera) methods often struggle with "temporal consistency"—they treat every frame as a new problem, leading to jittery estimates and a failure to handle brief occlusions. MSL-RAPTOR aims to bridge this gap: providing robust, smooth, and fast 6DoF tracking using only a standard RGB camera and onboard mobile processors.

Methods

The authors propose a two-stage architecture that splits the workload between a "Front-end" (vision) and a "Back-end" (estimation).

  1. The Front-end (Detection & Tracking):
    • YOLOv3: Used for initial object detection and semantic labeling (e.g., "this is a drone").
  2. SiamMask: Once an object is found, this visual tracker takes over. Unlike standard trackers that produce axis-aligned boxes, SiamMask produces angled bounding boxes. This is a key innovation: an angled box provides significantly more geometric information about an object's orientation than a simple upright square.
  3. The Back-end (The UKF):
    • The 2D box corners are fed into an Unscented Kalman Filter (UKF). The UKF maintains a Bayesian estimate of the object's 3D position, velocity, orientation, and angular velocity.
  4. The UKF uses a class-specific motion model. If the system knows it's tracking a drone, it can apply different physical constraints than if it were tracking a pedestrian.
  5. The Feedback Loop:
    • The system doesn't just pass data forward; it checks itself. It calculates the Mahalanobis distance between the predicted pose and the new measurement. If the distance is too high (indicating the tracker has drifted or the object is occluded), it triggers the YOLO detector to "reset" the system.

Datasets

The researchers evaluated MSL-RAPTOR on two distinct benchmarks:

  • NOCS-REAL275: A standard dataset for category-level 6D pose estimation containing common household objects (mugs, laptops, bottles). This allowed them to compare against state-of-the-art RGB-D methods.
  • Custom Aerial Robotics Scenario: The authors collected a new dataset consisting of 10,500 images (5,500 for training, 5,000 for testing) of a drone being tracked by another drone in flight. This dataset captures the high-speed motion and motion blur typical of real-world robotic applications.

Key Insights and Ideas

  • Angled Boxes Matter: The paper demonstrates that using the rotation of the 2D bounding box helps resolve pose ambiguity. While a 2D box can't perfectly define a 3D pose, an angled box narrows down the "manifold" of possible orientations much better than an axis-aligned one.
  • Filtering vs. Pure Inference: Most modern "pose from image" papers rely entirely on the neural network to output the pose. MSL-RAPTOR argues that classical probabilistic filters (like the UKF) are better at handling the "physics" of the problem, leading to smoother trajectories and better handling of noise.
  • Onboard Feasibility: By using a lightweight tracker (SiamMask) for most frames and only calling the heavy detector (YOLO) when necessary, the system maintains high frame rates on mobile hardware.

Key Results

  • Speed: On an NVIDIA Jetson TX2, MSL-RAPTOR runs at 8.7 Hz. This is 3x faster than the comparable SingleShotPose (SSP) method.
  • Accuracy in Flight: In the drone-to-drone tracking task, MSL-RAPTOR reduced median translation error by 66% and rotation error by 23% compared to SSP.
  • Monocular vs. Depth: On the NOCS-REAL275 benchmark, MSL-RAPTOR achieved performance comparable to methods that use depth sensors (RGB-D), despite only having access to 2D RGB data.

Limitations and Future Work

  • Depth Ambiguity: As with all monocular systems, estimating the exact distance (depth) along the camera's optical axis remains the largest source of error. The authors noted that depth estimates are significantly less accurate than in-plane (X-Y) translations.
  • Single Object Focus: The current implementation is optimized for tracking a single target. While the authors state the framework is extensible to multiple objects, doing so would increase the computational load on the Jetson TX2.
  • Initialization: The filter requires a rough initial guess for the 6D pose. While the authors use a simple heuristic (assuming the object is upright and aligned with the box), a more robust initialization for highly complex shapes could improve performance.

Summary updated

Abstract

Determining the relative position and orientation of objects in an environment is a fundamental building block for a wide range of robotics applications. To accomplish this task efficiently in practical settings, a method must be fast, use common sensors, and generalize easily to new objects and environments. We present MSL-RAPTOR, a two-stage algorithm for tracking a rigid body with a monocular camera. The image is first processed by an efficient neural network-based front-end to detect new objects and track 2D bounding boxes between frames. The class label and bounding box is passed to the back-end that updates the object’s pose using an unscented Kalman filter (UKF). The measurement posterior is fed back to the 2D tracker to improve robustness. The object’s class is identified so a class-specific UKF can be used if custom dynamics and constraints are known. Adapting to track the pose of new classes only requires providing a trained 2D object detector or labeled 2D bounding box data, as well as the approximate size of the objects. The performance of MSL-RAPTOR is first verified on the NOCS-REAL275 dataset, achieving results comparable to RGB-D approaches despite not using depth measurements. When tracking a flying drone from onboard another drone, it outperforms the fastest comparable method in speed by a factor of 3, while giving lower translation and rotation median errors by 66% and 23% respectively.
BibTeX
@inproceedings{ramtoula2020msl,
  author = {Benjamin Ramtoula and Adam Caccavale and Giovanni Beltrame and Mac Schwager},
  title = {{MSL-RAPTOR: A 6DoF Relative Pose Tracker for Onboard Robotic Perception}},
  booktitle = {ISER},
  year = {2020},
  url = {https://openreview.net/forum?id=iyfheeE3jG},
}

Loading related content...

Figure