Learning Visual Representation from Human Interactions

Kiana Ehsani, Daniel Gordon, Thomas Nguyen, Roozbeh Mottaghi, Ali FarhadiICLR 2021

TL;DR

This paper introduces a novel approach to learning visual representations by incorporating human interaction cues, specifically body part movements and gaze, alongside visual data. Their "muscly-supervised" method, trained on a new dataset of egocentric human interactions, significantly outperforms state-of-the-art visual-only self-supervised learning on a diverse set of downstream tasks.

What Problem Does This Paper Address?

The core challenge addressed is the limitation of current self-supervised visual representation learning methods, which primarily rely on visual data (images/videos) and often require large, curated datasets like ImageNet. While these methods have achieved impressive results, they may not fully capture the rich understanding humans derive from interacting with their environment and observing others. The authors hypothesize that incorporating human movement and attention (gaze) can provide stronger supervisory signals, leading to more generalizable and informative visual representations.

Methods

The paper proposes a "muscly-supervised" representation learning framework. The key innovation lies in jointly optimizing three objectives:

  1. Gaze Prediction: Predicting the human's focus of attention within the camera frame using a Huber loss.
  2. Body Part Movement Prediction: Predicting whether specific body parts are moving, framed as a binary classification task using binary cross-entropy loss. This simplifies the complex task of predicting precise movement magnitudes and directions.
  3. Auxiliary Visual Prediction: Employing an instance discrimination objective (similar to MoCo's InfoNCE loss) to learn invariant visual features. This objective encourages the network to map different augmentations of the same image closer together in the latent space.

These objectives are trained end-to-end using a CNN backbone (ResNet18) and sequence-to-sequence models (LSTMs) for gaze and movement prediction. The learned features from the backbone are then used for downstream tasks.

Datasets

The authors introduce a new dataset of human interactions. This dataset consists of:

  • Egocentric videos captured by a GoPro camera.
  • Simultaneous recording of body part movements using Inertial Measurement Units (IMUs) at 10 locations.
  • Gaze tracking data from a Tobii Pro2 eye-tracker.

The dataset comprises over 4,500 minutes of synchronized data from 35 participants engaged in daily activities, with no manual annotations provided.

For evaluation, they use the following standard datasets for target tasks:

  • Scene Classification: SUN397
  • Action Recognition: EPIC-KITCHENS
  • Dynamics Prediction: VIND
  • Walkable Surface Estimation: NYU DepthV2
  • Depth Estimation: NYU DepthV2

Key Insights and Ideas

  • Human interaction is a powerful supervisory signal: The paper strongly suggests that incorporating signals beyond pure visual input, like body movements and gaze, leads to significantly better visual representations.
  • Gaze grounds visual features: The authors hypothesize and demonstrate that gaze helps ground visual features to specific locations in an image, correlating well with semantic concepts and affordances.
  • Body movements encode temporal changes: The movement of body parts provides cues about the underlying causes of visual changes, aiding in understanding temporal dynamics.
  • Simplifying movement prediction is effective: Framing body part movement as a binary "moving/not moving" task, rather than precise regression, proves sufficient and less ambiguous for representation learning.
  • Diverse daily activities are rich for learning: The new dataset, collected during unconstrained daily routines, highlights the value of naturalistic interactions for learning generalizable representations.

Key Results

The "muscly-supervised" representation (using visual, movement, and gaze cues) consistently outperforms the state-of-the-art visual-only self-supervised baseline (MoCo trained on the same data) across all five target tasks:

  • Scene Classification: ~7% improvement over MoCo.
  • Action Recognition: ~3.5% improvement over MoCo.
  • Dynamics Prediction: ~1.3% improvement over MoCo.
  • Walkable Surface Estimation: The gaze-only variant achieved the highest accuracy, suggesting attention is crucial for this task.
  • Depth Estimation: The learned representation outperforms the baseline, with movement cues appearing particularly beneficial.

Ablation studies show that combining all three cues (visual, movement, attention) yields the best results.

Limitations and Future Work

  • Sensor Dependency: The current approach requires specialized sensors (IMUs, eye-trackers) for data collection, which limits scalability compared to purely visual methods.
  • Dataset Scalability: The authors acknowledge that scalability of their dataset requires further data collection.
  • Future Work: The paper suggests exploring more sophisticated ways to capture and leverage interaction signals, potentially moving towards more embodied AI systems. They also highlight the need for more diverse data collection to further improve scalability.

Summary updated

Abstract

Learning effective representations of visual data that generalize to a variety of downstream tasks has been a long quest for computer vision. Most representation learning approaches rely solely on visual data such as images or videos. In this paper, we explore a novel approach, where we use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. For this study, we collect a dataset of human interactions capturing body part movements and gaze in their daily lives. Our experiments show that our self-supervised representation that encodes interaction and attention cues outperforms a visual-only state-of-the-art method MoCo (He et al.,2020), on a variety of target tasks: scene classification (semantic), action recognition (temporal), depth estimation (geometric), dynamics prediction (physics) and walkable surface estimation (affordance).
BibTeX
@inproceedings{ehsani2021learning,
  author = {Kiana Ehsani and Daniel Gordon and Thomas Nguyen and Roozbeh Mottaghi and Ali Farhadi},
  title = {{Learning Visual Representation from Human Interactions}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure