Abstract
Human-Object Interaction (HOI) detection aims to identify triplets of <human, verb, object> within an image. While recent advancements in computer vision have significantly improved object detection, HOI remains challenging due to the complex spatial relationships and contextual nuances required to link humans to the objects they interact with. This paper identifies a critical performance gap in existing HOI models: distance sensitivity. Current models excel at recognizing close-range actions but struggle with distant interactions. To address this, we introduce the Spatially Differentiated Transformer (SDT), which incorporates a specialized Far-Near attention mechanism and a distance-weighted loss function. Our approach achieves new state-of-the-art results on the HICO-DET and V-COCO benchmarks.
Introduction
Human-Object Interaction (HOI) detection is essential for high-level scene understanding, with applications ranging from autonomous robotics to video surveillance. Despite the success of Transformer-based architectures in this domain, a systematic bias persists. In standard datasets like HICO-DET, the majority of labeled interactions occur at close range (e.g., holding a phone, sitting on a chair). Consequently, models tend to under-fit on distant interactions, such as flying a kite or throwing a ball across a field.
Furthermore, standard global attention mechanisms often suffer from "attention interference." When a model attempts to process a distant human-object pair, it can be distracted by nearby, non-interacting objects that occupy a larger portion of the visual field. This paper proposes that distance is not merely a spatial feature but a primary factor in interaction difficulty and data scarcity.
The Spatially Differentiated Transformer (SDT)
The SDT framework adopts a two-stage approach to decouple object localization from interaction reasoning, allowing for more specialized processing of spatial relationships.
Stage 1: Token Detection
The model utilizes a pre-trained Detection Transformer (DETR) to identify all humans and objects within the image. These entities are treated as "tokens" that serve as the input for the subsequent interaction recognition stage.
Stage 2: Interaction Recognition
The core innovation of SDT lies in how it processes these tokens to identify verbs. Three primary modules are introduced:
- Far-Near Distance Attention (FNDA): Unlike standard Multi-Head Self-Attention (MHSA) which treats all tokens equally regardless of distance, FNDA processes information in two distinct "glances." One glance focuses exclusively on distant tokens, while the other focuses on nearby tokens. This bifurcation prevents the signal from distant, smaller objects from being drowned out by the high-magnitude features of nearby objects.
- Distance-Aware (DA) Loss: To mitigate the imbalance in training data where distant interactions are rare, the authors designed a loss function that adaptively assigns higher weights to distant pairs. This forces the model to prioritize learning from these "hard" cases during the optimization process.
- Intra-Class Diversification (ICD): Distant objects are often occluded or represented by very few pixels. The ICD module employs a memory bank to "borrow" features from high-confidence, clear examples of the same object class, effectively augmenting the representation of small or degraded objects.
Experimental Results
The SDT model was evaluated on two primary HOI benchmarks, demonstrating superior performance over previous state-of-the-art (SOTA) methods.
HICO-DET
On the HICO-DET dataset, which contains 600 interaction classes, SDT achieved 32.97 mAP (mean Average Precision). This represents a significant improvement of approximately 1.3 mAP over the previous leading method, UPT. Qualitative analysis confirms that SDT is particularly effective at identifying interactions like "pulling a kite" or "launching a plane," where the distance between the human and the object is substantial.
V-COCO
On the V-COCO benchmark, SDT achieved 61.8 mAP in Scenario 1. This 2.8-point lead over previous SOTA models highlights the robustness of the spatially differentiated approach across different dataset distributions.
Discussion and Future Work
The success of SDT underscores the importance of treating distance as a first-class citizen in HOI detection. By filtering out local noise through FNDA and re-balancing the training signal via DA Loss, the model overcomes the inherent biases of standard attention mechanisms.
However, several challenges remain. As a two-stage method, SDT is inherently limited by the performance of the initial object detector; if a distant object is not detected in Stage 1, the interaction cannot be recovered. Additionally, the current model relies on 2D image coordinates to estimate distance. Future research could explore the integration of 3D depth estimation to provide a more accurate spatial context. Finally, addressing the "small object bottleneck" through advanced feature upsampling or specialized backbones remains a promising avenue for further improving distant HOI detection.
Summary updated