COBE: Contextualized Object Embeddings from Narrated Instructional Video

Gedas Bertasius, Lorenzo Torresani

Abstract

Many objects in the real world undergo dramatic variations in visual appearance. For example, a tomato may be red or green, sliced or chopped, fresh or fried, liquid or solid. Training a single detector to accurately recognize tomatoes in all these different states is challenging. On the other hand, contextual cues (e.g., the presence of a knife, a cutting board, a strainer or a pan) are often strongly indicative of how the object appears in the scene. Recognizing such contextual cues is useful not only to improve the accuracy of object detection or to determine the state of the object, but also to understand its functional properties and to infer ongoing or upcoming human-object interactions. A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms. Instead of relying on manually-labeled data for this task, we propose a new framework for learning Contextualized OBject Embeddings (COBE) from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. This enables the learning of an object representation where concepts relate according to a semantic language metric. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.

Introduction: Beyond Noun-Centric Object Detection

Traditional object detection systems are primarily "noun-centric." While a model might excel at identifying a "tomato," it often fails to distinguish between its various states—whether it is red or green, sliced, chopped, or being fried in a pan. In real-world scenarios, objects undergo dramatic visual transformations based on their state and the actions performed on them.

Variations in object appearance and state for an onion.

The challenge lies in the massive, long-tailed distribution of these states. Manually labeling every possible combination of object, adjective, and action (e.g., "peeling an onion" vs. "chopped onions") is practically impossible. COBE (Contextualized OBject Embeddings) addresses this by learning fine-grained contextual cues without relying on massive, manually-annotated datasets.

Methodology: Learning from Narrated Video

COBE utilizes a self-supervised approach that leverages the "free" supervision found in narrated instructional videos, such as cooking tutorials.

The COBE framework architecture for learning contextualized embeddings.

The framework consists of several key components:

  1. Pseudo-Labeling: A pretrained detector identifies objects in video frames. These detections are then aligned with transcribed narrations to find corresponding text (e.g., "egg").
  2. Contextualized Language Modeling: Instead of discrete labels, the entire narration (e.g., "...break an egg into a bowl...") is processed through a pretrained language model. This generates a contextualized word embedding that captures the noun alongside surrounding actions and co-occurring objects.
  3. Visual Embedding Prediction: The architecture modifies a standard Faster R-CNN. The traditional classification head is replaced with a head that predicts a continuous 1280-dimensional embedding.
  4. Contrastive Learning: The model is trained using Noise Contrastive Estimation (NCE) loss, aligning visual embeddings of detected objects with their corresponding language embeddings while distancing them from negative samples.

Key Insights and Results

By mapping images into a language space, COBE inherits the semantic "common sense" of the language model. This allows the model to understand that a "sliced tomato" is semantically closer to a "chopped tomato" than to an unrelated object like a "car."

Compositionality and Visual Analogies

The model demonstrates that visual concepts are compositional. Similar to word2vec's semantic arithmetic, COBE can perform visual analogies. For instance, by manipulating embeddings (e.g., "cutting dough" - "dough" + "fish"), the model can successfully retrieve images of "cutting fish."

Performance and Generalization

COBE significantly outperforms standard discrete detectors on predicting (noun, context) tuples. Notably, on the EPIC-Kitchens dataset, COBE achieved a mean Average Precision (mAP) of 10.3 for (noun, noun) pairs it had never encountered during training—a task where discrete detectors inherently score zero. Despite training on noisy, automatically-transcribed data from HowTo100M, the model generalizes effectively to manually-labeled datasets like EPIC-Kitchens and YouCook2.

Conclusion and Future Directions

While COBE shows great promise, it currently depends on pretrained detectors for initial pseudo-labels, which limits its vocabulary. Additionally, noisy narrations in transcribed speech remain a challenge. Future work aims to develop fully self-supervised systems that can discover objects and their states simultaneously, further scaling the training process to improve embedding quality.

Summary updated

Abstract

Many objects in the real world undergo dramatic variations in visual appearance. For example, a tomato may be red or green, sliced or chopped, fresh or fried, liquid or solid. Training a single detector to accurately recognize tomatoes in all these different states is challenging. On the other hand, contextual cues (e.g., the presence of a knife, a cutting board, a strainer or a pan) are often strongly indicative of how the object appears in the scene. Recognizing such contextual cues is useful not only to improve the accuracy of object detection or to determine the state of the object, but also to understand its functional properties and to infer ongoing or upcoming human-object interactions. A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms. Instead of relying on manually-labeled data for this task, we propose a new framework for learning Contextualized OBject Embeddings (COBE) from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. This enables the learning of an object representation where concepts relate according to a semantic language metric. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.
BibTeX
@inproceedings{bertasius2020cobe,
  author = {Gedas Bertasius and Lorenzo Torresani},
  title = {{COBE: Contextualized Object Embeddings from Narrated Instructional Video}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=J6mL2xP0B5},
}
Figure