TL;DR
This paper introduces a robust 6DoF object tracking pipeline using textured 3D scans, designed for challenging real-world conditions like motion blur and occlusions. They demonstrate its effectiveness in a remote live support AR application, offloading computation to an edge server to enable high-quality AR experiences on less powerful mobile devices.
What Problem Does This Paper Address?
Augmented Reality (AR) applications, especially in industrial settings like remote live support, rely heavily on accurately tracking the 6 Degrees of Freedom (6DoF) pose of objects in real-time. This allows virtual information to be precisely overlaid onto the real world. Traditional methods often use markers, which can be obtrusive and sensitive to occlusions. While markerless tracking methods like SLAM exist, they can struggle with object-specific AR content and real-world scale. Model-based tracking, using 3D object models, offers a promising alternative but faces challenges with initialization, robustness to varying conditions (illumination, motion blur, occlusions), and computational demands on mobile devices. This paper aims to address these limitations by providing a comprehensive, robust, and efficient 6DoF object tracking system suitable for demanding AR applications.
Methods
The core of the paper is a 6DoF object tracking pipeline built upon textured 3D scans of objects. The pipeline consists of several key stages:
- 3D Scanning and Registration: Objects are scanned using structured light to create realistic textured 3D models. This process is robust even for objects with poor natural texture. These models are then used to generate numerous rendered images from various viewpoints.
- Feature Learning: From these rendered images, salient "anchor points" are extracted using a detector like "Good Features To Track." These points, along with their 3D coordinates derived from depth information, form the basis for tracking.
- Initialization and Reinitialization: To handle cases where tracking is lost or the system starts, a two-stage approach is employed:
- ORB Initializer: Matches ORB features between the current frame and a library of pre-rendered object views to estimate an initial pose.
- ORB Reinitializer: A faster reinitialization mechanism that uses a "keyframe" (a recently captured frame with a known pose and extracted features) to quickly recover from tracking failures.
- Frame-to-Frame Tracking: This is the core tracking loop. It uses a combination of:
- Frame-to-Frame (F2F) Tracking: A standard KLT tracker to follow anchor points from the previous frame to the current one.
- Rendered-to-Frame (R2F) Tracking: To combat drift, the system renders the object from the previous frame's estimated pose and uses this rendered image as a reference for tracking, improving robustness.
- Pencil Filter: To enhance robustness to illumination changes, a "pencil filter" is applied. This filter enhances edges and normalizes local image intensities, significantly improving the similarity between rendered and real images under varying lighting.
- Remote Live Support Architecture: The tracking system is integrated into a remote live support application. This architecture leverages Mobile Edge Computing (MEC) to offload the computationally intensive tracking and rendering tasks from the mobile client device to a powerful edge server. This reduces latency and allows for complex AR experiences on less capable devices. The remote expert can then add 3D annotations to the live feed, which are accurately registered to the object's pose.
Datasets
Not explicitly discussed in terms of pre-existing datasets for training. The paper focuses on creating its own 3D models of objects through scanning and using these models to generate synthetic data (rendered images) for training and initialization. The evaluation uses a custom testbed with a Bosch Rexroth AG industrial component.
Key Insights and Ideas
- Textured 3D Scans are Key: Using textured 3D scans, rather than just CAD models, provides richer information for robust initialization and tracking, especially for objects where CAD models might not be available or accurate.
- Pencil Filter for Illumination Invariance: The simple yet effective "pencil filter" significantly improves tracking robustness under challenging lighting conditions by enhancing image features and normalization. This is a clever adaptation of an artistic filter for a practical computer vision task.
- Hybrid F2F/R2F Tracking for Drift Reduction: Combining standard frame-to-frame tracking with rendered-to-frame tracking is crucial for preventing the accumulation of drift over time, a common problem in visual tracking.
- Edge Computing for AR on Mobile: Offloading computationally heavy AR tasks (tracking, rendering) to an edge server is a viable and effective strategy to deliver high-quality AR experiences on resource-constrained mobile devices, enabling real-time performance with reduced latency.
- 3D Registration of Annotations: Accurately registering remote expert annotations in 3D space, based on the tracked object's pose, is critical for effective remote assistance and goes beyond simple 2D overlays.
Key Results
- Robust Tracking: The system demonstrates robust performance under challenging conditions, including scale changes, occlusions, motion blur, and illumination variations. Qualitative results show accurate placement of 3D annotations even in these scenarios.
- High Accuracy: Quantitative analysis shows high tracking accuracy with translational errors below 0.1 mm and angular errors below 0.01 rad in good lighting. Even in weak lighting, errors remain below 1 mm and 0.1 rad, respectively.
- Real-time Performance: On a desktop PC (acting as an edge server), the tracking achieves approximately 52 fps (19.12 ms per frame), which is sufficient for most AR applications.
- Mobile Device Limitations: On a mobile PC (Microsoft Surface Pro 4), the same tracking task takes significantly longer (44.75 ms per frame, ~22 fps), highlighting the need for offloading.
- Edge Computing Benefits: The remote live support application, utilizing edge computing, achieves low end-to-end latency (under 70 ms in wireless setups), enabling a responsive AR experience.
- Fast Initialization/Reinitialization: Initialization takes 80-160 ms, while reinitialization is very fast, averaging around 10-21 ms, allowing for quick recovery from tracking loss.
Limitations and Future Work
- Object Types: The paper acknowledges that objects that are highly reflective or completely textureless might be difficult to scan and track effectively with the current technology.
- Computational Resources: While the edge computing approach mitigates mobile device limitations, the overall system performance is still dependent on network bandwidth and latency.
- Future Work: The authors suggest further exploration of edge computing architectures and optimization of network communication for even lower latencies. They also implicitly point towards improving tracking for challenging object types.
Summary updated