Introduction
Detecting the "active object"—the item a person is currently manipulating—is a core challenge in computer vision, particularly for robotics, AR/VR, and activity recognition. Traditional object detectors often struggle in these scenarios because hands frequently occlude the very objects they touch. This paper proposes a shift from independent detection to a hand-conditioned approach using Relational Box Fields (RBF).
The Problem: Occlusion and Fragile Linking
Standard methods typically detect hands and objects separately and then attempt to link them. This "detect-then-match" strategy is fragile; if an object is significantly hidden by a hand, the detector may fail to find it at all, leaving no object to link to the hand. The authors argue that a hand's appearance—its grip, finger positions, and orientation—contains vital clues about the object's location and scale, even when the object itself is mostly invisible.
Methodology: Relational Box Fields and Sequential Voting
The proposed solution involves two main components: pixel-wise voting and iterative refinement.
Relational Box Fields (RBF)
Instead of a single global prediction, the model generates a "field" of bounding box hypotheses. Every pixel within a detected hand region predicts its own version of the active object's bounding box (center, height, width) along with a confidence score. This allows the model to aggregate subtle visual cues from across the hand's surface.
Sequential Refinement via MDP
The detection process is framed as a Markov Decision Process (MDP). An initial estimate is derived from the hand's pixels. The model then looks at the pixels within that estimated region to "vote" again, refining the box's parameters. This iterative process allows the model to converge on a precise bounding box.
Robust Voting and RL Training
To handle the thousands of pixel-level predictions, a weighted voting mechanism identifies the majority consensus, making the system robust to outliers and noisy background pixels. To prevent "drift" (covariate shift) during sequential steps, the model is trained using Reinforcement Learning (RL), which teaches it to recover from early estimation errors and maintain stability.
Experimental Results
The model was evaluated on two challenging datasets: 100DOH (100 Degrees of Hands) and MECCANO (an industrial assembly dataset).
- Significant Performance Gains: The method achieved an 8% AP50 improvement on 100DOH and a staggering 30% improvement on the egocentric MECCANO dataset compared to previous state-of-the-art baselines.
- Robustness to Occlusion: The system maintained high recall in scenarios where objects were almost entirely covered by hands, where traditional detectors typically fail.
- Efficiency: The architecture is optimized for real-time applications, running at approximately 18 frames per second (FPS) on a standard desktop GPU.
Conclusion
By leveraging the hand as a visual proxy for the object it holds, Relational Box Fields overcome the limitations of traditional detectors in high-occlusion environments. While the method depends on a reliable initial hand detection, its ability to self-correct through sequential voting represents a significant advancement in understanding human-object interactions.
Summary updated