TL;DR
LIGHTEN is a new hierarchical framework that masters Human-Object Interaction (HOI) detection in videos using only raw visual data. By combining Graph Convolutional Networks (GCNs) with a two-tier temporal RNN structure, it outperforms previous methods that relied on "crutches" like 3D pose or depth maps, setting a new state-of-the-art for video-based HOI.
What Problem Does This Paper Address?
Understanding how humans interact with objects—like "pouring water" or "opening a fridge"—is a cornerstone of computer vision. While image-based HOI has seen huge leaps, video-based HOI is significantly more challenging. In a video, the orientation of objects changes, and many actions (like "opening" vs. "closing" a jar) look identical in a single frame; they can only be distinguished by looking at the sequence of movements.
Existing video HOI models often rely on hand-crafted features derived from ground-truth data, such as 3D human poses, object centroids, or depth maps. This makes them brittle: if you don't have a depth sensor or a perfect pose estimator, the model fails. Furthermore, most models only look at temporal cues at a coarse "segment" level, missing the fine-grained, frame-by-frame nuances that define an interaction.
Methods
The authors propose LIGHTEN (Learning Interactions with Graph and Hierarchical TEmporal Networks), an end-to-end pipeline that operates in three distinct stages:
- Spatial Subnet (The "Where"): For every frame, the model treats the human and objects as nodes in a graph. It uses a Graph Convolutional Network (GCN) to learn the spatial relationships between them. Crucially, it uses an adaptive adjacency matrix that learns which interactions matter most for a specific input, rather than relying on a fixed structure.
- Frame-level Temporal Subnet (The "How"): The spatial features from the GCN are fed into a bidirectional RNN. This stage captures the fine-grained motion within a single video segment (e.g., the specific arc of a hand during a "reach").
- Segment-level Temporal Subnet (The "Context"): Interactions don't happen in a vacuum. "Placing" usually follows "moving." This third stage uses another RNN to model the dependencies between consecutive segments. It employs an attention mechanism to decide which frames within a segment are most representative before passing that information up the hierarchy.
The entire system is trained on pure RGB visual features extracted from a standard ResNet-50 backbone, making it highly generalizable to "in-the-wild" videos.
Datasets
- CAD-120: The primary benchmark for video HOI. It contains 120 RGB-D videos of daily activities. Unlike previous researchers, the authors of LIGHTEN ignored the depth and 3D pose data, using only the RGB frames and 2D bounding boxes.
- V-COCO: A subset of MS-COCO used for image-based HOI. The authors tested LIGHTEN here to demonstrate that their architecture can "degenerate" gracefully to handle static images by setting the number of frames to one.
Key Insights and Ideas
- Ditching the Crutches: The most significant "aha" moment is that learned visual features can actually outperform hand-crafted 3D features. By providing a sufficiently rich hierarchical structure, the model learns the spatial and temporal cues it needs directly from the pixels.
- Hierarchical is Essential: Modeling time at two scales (frame-level and segment-level) is what allows the model to disambiguate confusingly similar actions. The segment-level context acts as a "sanity check" for the frame-level predictions.
- Adaptive Graphs: Since the number of objects in a scene varies (you might be interacting with one bowl or three different ingredients), the use of an adaptive GCN allows the model to dynamically adjust its focus without needing a fixed-size input.
Key Results
LIGHTEN achieved impressive results on the CAD-120 dataset, even without using the 3D/depth data that its competitors relied on:
- Sub-activity Detection: Achieved an 88.9% F1 score, outperforming the previous best (GPNN) which sat at 88.9% but required hand-crafted features.
- Object Affordance Detection: Reached 92.6% F1 score, a significant jump over the previous state-of-the-art (91.1%).
- Anticipation: The model also excels at predicting future interactions, achieving 76.4% in sub-activity anticipation, significantly higher than the 75.6% of previous methods.
Limitations and Future Work
While LIGHTEN is a powerhouse for video, its performance on static images (V-COCO) is competitive but not state-of-the-art (38.28 mAP vs. 52.0 mAP for specialized image models). The authors note that in the absence of temporal cues, their spatial GCN is "shallower" than models specifically designed for single-image reasoning.
Future work could involve integrating more sophisticated spatial reasoning modules that don't rely on temporal context, or extending the model to handle multi-human interactions, which would significantly increase the complexity of the graph structure.
Summary updated