TL;DR
Real-world objects change appearance drastically based on their state and context (e.g., a whole tomato vs. a chopped tomato in a pan). This paper introduces Contextualized OBject Embeddings (COBE), a framework that learns to recognize fine-grained object states and interactions from narrated instructional videos without requiring manual, fine-grained labels. By aligning visual detections with contextualized word embeddings from speech transcripts, COBE enables powerful zero-shot and few-shot detection of complex object states.
What Problem Does This Paper Address?
Standard object detection models are typically trained on datasets with coarse, discrete noun categories (e.g., "tomato", "egg", "pan"). However, this noun-centric ontology fails to capture the dramatic visual variations objects undergo in the real world.
An egg can be whole, cracked, fried, or mixed in a bowl. Similarly, an onion looks entirely different when it is being peeled, chopped, or cooked into a sauce. Training a fully-supervised model to recognize all these different states and contexts is incredibly difficult because the distribution of object states is open-ended and long-tailed. Collecting exhaustive bounding-box annotations for every possible (object, state) combination is practically impossible.
The authors tackle this gap by asking: Can we learn these rich, fine-grained object states using the "free" supervision provided by the natural language narrations in instructional videos?
Methods
Instead of training a detector to output a discrete class label, the authors train a model to output a continuous semantic vector that aligns with a language model.
- Pseudo-Bounding Boxes: Because their training video dataset (HowTo100M) lacks bounding boxes, they use a standard object detector pretrained on the EPIC-Kitchens dataset to generate "pseudo" ground-truth bounding boxes for coarse object categories.
- Contextualized Language Model: They process the automatically transcribed speech (ASR) corresponding to a video frame using a Transformer-based language model (like BERT). If the narration says "...break an egg into a bowl...", the model extracts a rich, contextualized word embedding for the word "egg" that inherently encodes the action ("break") and the context ("bowl").
- Visual Detection Model (COBE): They modify a Faster R-CNN architecture. Instead of a standard classification head, the network features a COBE branch that predicts a continuous dense vector for every detected Region of Interest (RoI).
- Contrastive Learning: The model is trained using a Noise Contrastive Estimation (NCE) loss. It forces the predicted visual embedding of the object to be highly similar to its corresponding contextualized word embedding, and dissimilar to negative word embeddings.
Datasets
- Training: HowTo100M_BB, a newly curated subset of the HowTo100M instructional video dataset. It contains ~559K frames and 1.1M pseudo-bounding boxes spanning 154 object categories.
- Evaluation: EPIC-Kitchens (EK), YouCook2_BB, and a held-out test set of HowTo100M_BB.
- New Benchmark Task: The authors introduce the task of contextualized object detection, where the model must predict a bounding box and a
(noun, context)tuple (e.g.,(onion, chopped)or(pan, mushrooms)).
Key Insights and Ideas
- Continuous Spaces Beat Discrete Labels: By mapping visual objects into a continuous language embedding space rather than forcing them into discrete categories, the model can handle the long-tailed, open-ended nature of real-world object states.
- Instructional Videos are Goldmines: The way humans narrate tutorials naturally describes object states, appearances, and actions. This provides a massive, untapped source of fine-grained supervisory signals.
- Visual Compositionality: Because COBE aligns visual features with a modern language model, the visual embeddings inherit the semantic and compositional structure of language. The authors demonstrate that you can perform vector arithmetic with images (e.g.,
[image of cutting dough] - [image of dough] + [image of fish] = [image of cutting fish]).
Key Results
- Contextualized Detection: COBE significantly outperforms a "Tuple Faster R-CNN" baseline (a standard detector trained directly on discrete tuple labels) across HowTo100M, EPIC-Kitchens, and YouCook2 datasets.
- Beating Massive Supervised Baselines: On EPIC-Kitchens, COBE (trained only on HowTo100M) was compared against baselines combining Faster R-CNN with state-of-the-art classifiers trained on massive manually labeled datasets (ImageNet, Kinetics, IG65M). COBE crushed them. For example, on
(noun, noun)detection, COBE achieved 32.4 mAP compared to the IG65M baseline's 8.4 mAP. - Zero-Shot and Few-Shot Learning: When tested on entirely unseen tuple categories in EPIC-Kitchens, COBE achieved 10.3 mAP zero-shot and 25.5 mAP few-shot (5 examples) for
(noun, noun)combinations, proving its ability to generalize concepts to novel classes. - Standard Object Detection Boost: Pretraining COBE on HowTo100M before fine-tuning it on EPIC-Kitchens for standard object detection yielded 22.6 mAP, nearly double the 12.7 mAP achieved by training solely on EPIC-Kitchens.
Limitations and Future Work
- Reliance on a Pretrained Detector: The most glaring limitation is that the pipeline relies on a pretrained object detector (trained on EPIC-Kitchens) to generate the initial pseudo-bounding boxes. This restricts the system to learning contexts only for the predefined set of coarse object classes recognized by that initial detector.
- Future Work: The authors acknowledge this bottleneck and suggest that future work should focus on extending this framework to be fully self-supervised—learning contextualized object detection directly from instructional videos without the crutch of a pretrained visual model. However, they note this is highly challenging due to the inherent noise in automatically transcribed video captions.
Summary updated