Introduction
Human-Object Interaction (HOI) detection involves identifying triplets such as <person, kick, ball> within a scene. While progress has been made in static images, video HOI detection remains a significant challenge due to temporal dynamics and the "open-set" problem. Actions like "pushing" and "pulling" may appear identical in a single frame, requiring temporal context for differentiation. Furthermore, real-world interactions are infinite, yet most models are restricted to a "closed set" of predefined actions.
To address these issues, the authors propose ACoLP (Action-centric Chain-of-Look Prompting). This framework shifts the focus from identifying nouns (human and object) to prioritizing the action (the verb) as the primary carrier of meaning. Inspired by the "Chain-of-Thought" prompting in Large Language Models (LLMs), ACoLP introduces a "Chain-of-Look" (CoL) scheme to break down visual reasoning into intermediate steps.
Methodology: The ACoLP Framework
The ACoLP framework utilizes two specialized networks to perform multi-step visual reasoning, leveraging pre-trained Vision-Language Models (VLMs) like CLIP to handle unseen actions through semantic relationships.
Visual-Semantic Reasoning (VSR) Network
The VSR network refines HOI prompts by establishing context. It first generates a global image caption using BLIP to understand the general scene (CaptionHOI Prompting). Subsequently, it incorporates local visual features of the human and object to tailor the prompt to the specific video clip (VideoHOI Prompting).
Spatio-Temporal Reasoning (STR) Network
The STR network focuses on the temporal evolution of the interaction. An Action Prompting module aligns visual combinations with action labels. These are then processed by a Dynamic Graph Neural Network (GNN) to track how interactions evolve across frames, capturing the "flow" of the action.
Experiments and Results
The model was evaluated on the VidHOI and CAD120 benchmarks. To test open-set capabilities, subsets were created where 20% or 50% of action classes were withheld during training.
Key Results
ACoLP established a new state-of-the-art (SOTA) across several metrics:
- Open-Set Performance: On VidHOI with 20% unseen classes, ACoLP achieved 19.23 mAP, surpassing the previous best of 15.86.
- Closed-Set Performance: ACoLP led with 28.27 mAP on VidHOI and a 94.9 F1 score on CAD120.
- Temporal Reasoning: The model excelled in "temporal HOIs" (actions requiring motion), outperforming the next best method by over 3% mAP.
Discussion and Conclusion
The success of ACoLP highlights that centering models on action prompts rather than just object features allows for better generalization to unseen interactions. The "Chain-of-Look" approach provides a robust internal representation by building from global semantic anchors to local details.
However, the framework's reliance on multiple heavy pre-trained models (Faster R-CNN, BLIP, and CLIP) results in high computational costs, making real-time application challenging. Additionally, performance remains dependent on the accuracy of the initial object detector. Future research may focus on end-to-end training or distillation techniques to improve efficiency while maintaining reasoning depth.
Summary updated