VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions

Oytun Ulutan, A. S. M. Iftekhar, Bangalore S. Manjunath

Abstract

Comprehensive visual understanding requires detection frameworks that can effectively learn and utilize object interactions while analyzing objects individually. This is the main objective in Human-Object Interaction (HOI) detection task. In particular, relative spatial reasoning and structural connections between objects are essential cues for analyzing interactions, which is addressed by the proposed Visual-Spatial-Graph Network (VSGNet) architecture. VSGNet extracts visual features from the human-object pairs, refines the features with spatial configurations of the pair, and utilizes the structural connections between the pair via graph convolutions. The performance of VSGNet is thoroughly evaluated using the Verbs in COCO (V-COCO) and HICO-DET datasets. Experimental results indicate that VSGNet outperforms state-of-the-art solutions by 8% or 4 mAP in V-COCO and 16% or 3 mAP in HICO-DET.

Introduction

Detecting Human-Object Interaction (HOI) is a critical step toward deep scene understanding. Unlike standard object detection, HOI requires identifying a human, an object, and the specific action (verb) that connects them. For example, a model must distinguish between a person "holding" a bicycle and a person "riding" a bicycle.

Current research in HOI detection often overlooks two vital sources of information:

  1. Refined Spatial Cues: While the relative positions of humans and objects are often used as raw features, they are rarely used to dynamically refine visual features. The spatial layout of a pair provides a strong prior for the type of interaction occurring.
  2. Structural Context: Interactions do not occur in isolation. A scene often contains multiple humans and objects with complex, overlapping relationships. Treating human-object pairs as independent entities ignores the global structural context of the scene.
Conceptual overview of the Visual, Spatial, and Graph components in VSGNet.

To address these gaps, we introduce VSGNet (Visual-Spatial-Graph Network), which explicitly models spatial and structural relationships to enhance HOI detection.

Methods

VSGNet employs a three-pronged architecture to process visual, spatial, and structural information simultaneously.

The VSGNet architecture, integrating visual features, spatial attention, and graph convolutional networks.

Visual Branch

The visual branch serves as the foundation, utilizing a ResNet-152 backbone. It employs Region of Interest (ROI) pooling to extract features for the human, the object, and the global scene context. These features provide the raw visual evidence for potential interactions.

Spatial Attention Branch

A key innovation of VSGNet is the use of spatial configuration as an attention mechanism. The model generates two-channel binary maps representing the bounding boxes of the human and the object. Convolutions are applied to these maps to produce spatial attention weights. These weights are then multiplied element-wise with the visual features, effectively "filtering" the visual data to focus on regions consistent with the observed spatial layout.

Graph Convolutional Branch

To capture the structural context of the scene, VSGNet treats humans and objects as nodes in a graph. Unlike traditional GCNs that rely on visual similarity for edge definition, VSGNet uses interaction proposal scores—the estimated likelihood of an interaction—to weight the edges. This allows the network to update node features based on the most relevant structural connections in the scene.

Experimental Results

The performance of VSGNet was evaluated on two major HOI benchmarks: V-COCO and HICO-DET.

  • V-COCO: VSGNet achieved 51.76 mAP, representing an 8% improvement over previous state-of-the-art methods.
  • HICO-DET: The model reached 19.80 mAP, a significant 16% improvement.

Ablation studies confirmed that the spatial attention branch provided the most substantial performance gain (+3 mAP), highlighting that spatial reasoning is a critical bottleneck in current HOI detection pipelines. Qualitatively, the model demonstrated robustness in cluttered scenes and with partially occluded objects, where spatial and structural cues compensated for degraded visual information.

Conclusion

VSGNet demonstrates that explicitly modeling spatial attention and structural graphs significantly improves Human-Object Interaction detection. By using spatial layouts to refine visual features and interaction-driven graphs to capture scene context, the model sets a new standard for the field. Future work may involve integrating human pose estimation to provide even finer-grained spatial cues and addressing the dependency on the underlying object detector's performance.

Summary updated

Abstract

Comprehensive visual understanding requires detection frameworks that can effectively learn and utilize object interactions while analyzing objects individually. This is the main objective in Human-Object Interaction (HOI) detection task. In particular, relative spatial reasoning and structural connections between objects are essential cues for analyzing interactions, which is addressed by the proposed Visual-Spatial-Graph Network (VSGNet) architecture. VSGNet extracts visual features from the human-object pairs, refines the features with spatial configurations of the pair, and utilizes the structural connections between the pair via graph convolutions. The performance of VSGNet is thoroughly evaluated using the Verbs in COCO (V-COCO) and HICO-DET datasets. Experimental results indicate that VSGNet outperforms state-of-the-art solutions by 8% or 4 mAP in V-COCO and 16% or 3 mAP in HICO-DET.
BibTeX
@inproceedings{ulutan2020vsgnet,
  author = {Oytun Ulutan and A. S. M. Iftekhar and Bangalore S. Manjunath},
  title = {{VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=NZi2yreblpA},
}
Figure