Sequential Voting with Relational Box Fields for Active Object Detection

Qichen Fu, Xingyu Liu, Kris M. Kitani

Introduction

Detecting the "active object"—the item a person is currently manipulating—is a core challenge in computer vision, particularly for robotics, AR/VR, and activity recognition. Traditional object detectors often struggle in these scenarios because hands frequently occlude the very objects they touch. This paper proposes a shift from independent detection to a hand-conditioned approach using Relational Box Fields (RBF).

Comparison of traditional detect-then-match methods versus the proposed hand-conditioned active object detection.

The Problem: Occlusion and Fragile Linking

Standard methods typically detect hands and objects separately and then attempt to link them. This "detect-then-match" strategy is fragile; if an object is significantly hidden by a hand, the detector may fail to find it at all, leaving no object to link to the hand. The authors argue that a hand's appearance—its grip, finger positions, and orientation—contains vital clues about the object's location and scale, even when the object itself is mostly invisible.

Methodology: Relational Box Fields and Sequential Voting

The proposed solution involves two main components: pixel-wise voting and iterative refinement.

Relational Box Fields (RBF)

Instead of a single global prediction, the model generates a "field" of bounding box hypotheses. Every pixel within a detected hand region predicts its own version of the active object's bounding box (center, height, width) along with a confidence score. This allows the model to aggregate subtle visual cues from across the hand's surface.

Sequential Refinement via MDP

The detection process is framed as a Markov Decision Process (MDP). An initial estimate is derived from the hand's pixels. The model then looks at the pixels within that estimated region to "vote" again, refining the box's parameters. This iterative process allows the model to converge on a precise bounding box.

The system pipeline showing the generation of box fields and the iterative refinement process.

Robust Voting and RL Training

To handle the thousands of pixel-level predictions, a weighted voting mechanism identifies the majority consensus, making the system robust to outliers and noisy background pixels. To prevent "drift" (covariate shift) during sequential steps, the model is trained using Reinforcement Learning (RL), which teaches it to recover from early estimation errors and maintain stability.

Experimental Results

The model was evaluated on two challenging datasets: 100DOH (100 Degrees of Hands) and MECCANO (an industrial assembly dataset).

  • Significant Performance Gains: The method achieved an 8% AP50 improvement on 100DOH and a staggering 30% improvement on the egocentric MECCANO dataset compared to previous state-of-the-art baselines.
  • Robustness to Occlusion: The system maintained high recall in scenarios where objects were almost entirely covered by hands, where traditional detectors typically fail.
  • Efficiency: The architecture is optimized for real-time applications, running at approximately 18 frames per second (FPS) on a standard desktop GPU.

Conclusion

By leveraging the hand as a visual proxy for the object it holds, Relational Box Fields overcome the limitations of traditional detectors in high-occlusion environments. While the method depends on a reliable initial hand detection, its ability to self-correct through sequential voting represents a significant advancement in understanding human-object interactions.

Summary updated

Abstract

A key component of understanding hand-object interactions is the ability to identify the active object -- the object that is being manipulated by the human hand. In order to accurately localize the active object, any method must reason using information encoded by each image pixel, such as whether it belongs to the hand, the object, or the background. To leverage each pixel as evidence to determine the bounding box of the active object, we propose a pixel-wise voting function. Our pixel-wise voting function takes an initial bounding box as input and produces an improved bounding box of the active object as output. The voting function is designed so that each pixel inside of the input bounding box votes for an improved bounding box, and the box with the majority vote is selected as the output. We call the collection of bounding boxes generated inside of the voting function, the Relational Box Field, as it characterizes a field of bounding boxes defined in relationship to the current bounding box. While our voting function is able to improve the bounding box of the active object, one round of voting is typically not enough to accurately localize the active object. Therefore, we repeatedly apply the voting function to sequentially improve the location of the bounding box. However, since it is known that repeatedly applying a one-step predictor (i.e., auto-regressive processing with our voting function) can cause a data distribution shift, we mitigate this issue using reinforcement learning (RL). We adopt standard RL to learn the voting function parameters and show that it provides a meaningful improvement over a standard supervised learning approach. We perform experiments on two large-scale datasets: 100DOH and MECCANO, improving AP50 performance by 8% and 30%, respectively, over the state of the art.
BibTeX
@inproceedings{fu2021sequential,
  author = {Qichen Fu and Xingyu Liu and Kris M. Kitani},
  title = {{Sequential Voting with Relational Box Fields for Active Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
}
Figure