EXOT: Exit-aware Object Tracker for Safe Robotic Manipulation of Moving Object

Hyunseo Kim, Hye Jung Yoon, Minji Kim, Dong-Sig Han, Byoung-Tak Zhang

TL;DR

Researchers have developed EXOT (Exit-aware Object Tracker), a new framework that prevents robots from "hallucinating" targets that are no longer there. By integrating an out-of-distribution (OOD) classifier into a Transformer-based tracker, EXOT allows robots to safely pause manipulation tasks when a moving object exits the camera's view, achieving a 38% improvement in exit-detection performance over state-of-the-art baselines.

What Problem Does This Paper Address?

In traditional robotic manipulation, a robot often calculates an object's position once and then moves "blindly" to execute a grasp or placement. This works in controlled labs, but fails in dynamic environments—like a busy kitchen or a factory floor—where objects move, disappear, or are blocked.

Current single-object trackers are designed to keep following a target, but they have a fatal flaw for robotics: when the target leaves the frame (an "exit" event), the tracker often latches onto background noise or similar-looking objects. For a robot, this "false positive" is dangerous; it might try to place a delicate item on an empty conveyor belt or, worse, collide with a human. The authors argue that for safe operation, a robot must be "exit-aware"—it needs to know exactly when it has lost sight of its target.

Fig. 1: An overview of the problem setting. A robot with a hand camera tracks down the target object (highlighted with green) on a moving conveyor-belt. Target objects are not evenly positioned on the conveyor-belt, so empty spots come up frequently. The robot recognizes an abnormal situation (exit) and does not try placing when the target object is not visible on the camera. When the target object is visible on the hand cam, the robot places the object it was holding. A camera frame with the target object is in-distribution (ID) and the frame without the target object is out-of-distribution (OOD).

Methods

The researchers built EXOT on top of STARK, a high-performance Transformer-based long-term tracker. Their key innovation is the addition of a dedicated OOD Score Head.

  1. The Architecture: EXOT uses a ResNet backbone to extract features from a search region and a target template. These are processed by a Transformer encoder-decoder to predict a bounding box.
  2. The OOD Head: Inspired by the "Generalized ODIN" technique, this head doesn't just classify the object; it explicitly models the domain probability. It splits the classification into two functions: one predicting the joint probability of the class and the domain ($h_i(x)$), and another predicting the domain probability itself ($g(x)$). The final OOD score is the ratio of these two, which helps the model distinguish between the target object (in-distribution) and the background (out-of-distribution).
  3. Perturbation for Clarity: During inference, the system applies a small, calculated perturbation to the input image. This "pushes" in-distribution and out-of-distribution samples further apart in the feature space, making the exit signal much clearer.
Fig. 2: An illustration of the proposed tracker, EXOT. At the start of training, a search image and an initial template are processed with the backbone network and concatenated. Then those are fed into the Transformer Encoder and Decoder sequentially. Three head networks (OOD score head, bounding box (bbox) prediction head, and score head) process inputs described in the diagram and output a classification confidence score, bbox coordinates, and a template update score, respectively. Using the information, a dynamic template is cropped from the search image and used as an input in the next step. The inside networks of OOD score head are described on the right. Further details are explained in Eq. 2.

Datasets

The authors evaluated EXOT on two primary datasets:

  • TREK-150: A benchmark for first-person (egocentric) visual object tracking, featuring complex kitchen environments where objects frequently exit the frame.
  • RMOT-223 (New): A custom dataset collected using a UR5e robot arm. It includes 223 videos of pick-and-place activities, captured via both automated operation and human teleoperation. This dataset specifically addresses the "gap" in robotic-view tracking data.

Key Insights and Ideas

  • Synergy between Tracking and Classification: The authors found that training the bounding box prediction and the OOD classification simultaneously (1-stage training) actually improved performance. The classification task helps the model "concretize" image features, making the localization more robust.
  • Raw Features Matter: When testing different inputs for the OOD head (like attention weights or query features), the authors discovered that using the raw backbone features ($f_x$) from the search image yielded the best results. The original image information is more useful for OOD detection than processed attention maps.
  • Safety First (Low FPR): In robotics, a False Positive (thinking the object is there when it isn't) is much worse than a False Negative. EXOT is tuned to minimize the False Positive Ratio (FPR), ensuring the robot defaults to a "safe" state (pausing) when uncertain.

Key Results

EXOT significantly outperformed the baseline STARK tracker in recognizing object exits:

  • Exit-Aware Performance: EXOT showed a 38% higher performance in exit detection compared to the baseline.
  • Quantitative Success: On the RMOT-223 dataset, EXOT achieved a significantly lower False Positive Ratio (0.71 vs 1.00 for the baseline in some configurations), meaning it was far less likely to mistake the background for the target.
  • Real-World Validation: In a "conveyor-belt sushi" experiment, a robot equipped with EXOT successfully paused when a customer took a plate, waiting for the next plate to arrive. The baseline robot, however, failed by attempting to place sushi on the empty moving belt because it "tracked" the dark background of the belt as the plate.

Limitations and Future Work

The primary limitation noted is the OOD threshold ($\phi$). Currently, this threshold needs to be examined and set per dataset type, as OOD score distributions can vary.

The authors suggest that future work could involve online learning to allow the robot to adapt its threshold dynamically as it operates in new environments. Additionally, while EXOT handles "exits" well, integrating it into more complex multi-step manipulation pipelines remains an open area for exploration.

Summary updated

Abstract

Current robotic hand manipulation narrowly operates with objects in predictable positions in limited environments. Thus, when the location of the target object deviates severely from the expected location, a robot sometimes responds in an unexpected way, especially when it operates with a human. For safe robot operation, we propose the EXit-aware Object Tracker (EXOT) on a robot hand camera that recognizes an object's absence during manipulation. The robot decides whether to proceed by examining the tracker's bounding box output containing the target object. We adopt an out-of-distribution classifier for more accurate object recognition since trackers can mistrack a background as a target object. To the best of our knowledge, our method is the first approach of applying an out-of-distribution classification technique to a tracker output. We evaluate our method on the first-person video benchmark dataset, TREK-150, and on the custom dataset, RMOT-223, that we collect from the UR5e robot. Then we test our tracker on the UR5e robot in real-time with a conveyor-belt sushi task, to examine the tracker's ability to track target dishes and to determine the exit status. Our tracker shows 38% higher exit-aware performance than a baseline method. The dataset and the code will be released at https://github.com/hskAlena/EXOT.
BibTeX
@inproceedings{kim2023exot,
  author = {Hyunseo Kim and Hye Jung Yoon and Minji Kim and Dong-Sig Han and Byoung-Tak Zhang},
  title = {{EXOT: Exit-aware Object Tracker for Safe Robotic Manipulation of Moving Object}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160481},
}
Figure