Abstract
Robotic manipulation in dynamic environments requires visual trackers that can reliably detect when an object leaves the field of view. Traditional trackers often "hallucinate" targets on background textures during exit events, leading to dangerous robotic failures. We present EXOT (Exit-aware Object Tracker), a framework that integrates an Out-of-Distribution (OOD) classifier into a Transformer-based tracker. By evaluating whether a frame is "in-distribution" (object present) or "out-of-distribution" (object absent), EXOT achieves a 38% improvement in exit-aware performance, ensuring safer robotic pick-and-place operations in unpredictable settings.
The Challenge of Dynamic Environments
In standard robotic manipulation, systems often calculate an object's position once and move blindly to execute a task. While effective in static labs, this approach fails on factory conveyor belts or in busy kitchens where objects move or disappear.
State-of-the-art visual trackers are typically designed to maintain tracking at all costs. When an object exits the frame, these trackers frequently latch onto background elements that vaguely resemble the target. For a robot, this "hallucination" is hazardous; it may attempt to grasp empty space or collide with the environment. EXOT addresses this by providing a reliable signal to the robot to stop or wait when the target is no longer present.
Methodology: Integrating OOD Detection
EXOT is built upon STARK, a high-performance Transformer-based long-term tracker. The core innovation is the addition of an OOD Score Head, inspired by the Generalized ODIN technique. This module classifies the current camera frame as either in-distribution or out-of-distribution.
System Architecture
- Backbone & Transformer: A ResNet backbone extracts features, while a Transformer encoder-decoder relates the search region to the initial target template.
- Three-Pronged Output: The model generates a bounding box for localization, a template update score for appearance changes, and an OOD score to confirm the object's presence.
- Input Perturbation: During inference, the system applies small, gradient-based perturbations to the input image. This technique further separates the signals for "present" and "absent" states, making the OOD classification more robust.
Evaluation and Key Results
The researchers evaluated EXOT using two primary benchmarks: TREK-150 (egocentric kitchen activities) and a new dataset, RMOT-223 (Robot Moving Object Tracking). RMOT-223 consists of 223 videos collected from a UR5e robot arm, capturing both automated and teleoperated sequences.
Performance Gains
- Exit-Awareness: EXOT demonstrated a 38% higher exit-aware performance compared to the baseline STARK tracker.
- Conservative Robotics: The system achieved significantly lower False Positive Ratios (FPR), prioritizing safety by ensuring the robot waits for a clear target rather than acting on erroneous background data.
- Real-World Application: In a "conveyor-belt sushi" demonstration, the EXOT-equipped robot correctly identified missing plates and waited for new targets to arrive, whereas the baseline tracker attempted to place items on the empty belt surface.
Conclusion and Future Work
EXOT demonstrates that treating classification and tracking as synergistic tasks significantly enhances robotic safety. By using raw backbone features for the OOD head, the system maintains a better context of the search region to determine true exits.
Currently, the system relies on a manually set threshold parameter ($\phi$) for OOD detection. Future research will focus on implementing online learning to allow the tracker to adapt this threshold dynamically, enabling robots to learn the specific characteristics of new workspaces on the fly.
Summary updated