Learning to Learn Words from Narrated Video

Dídac Surís, Dave Epstein, Heng Ji, Shih-Fu Chang, Carl VondrickarXiv 2019 · OpenReview

TL;DR

This paper introduces a novel meta-learning framework called EXPERT that learns how to acquire new words from visual scenes, rather than just learning word representations directly. This approach allows machines to learn new vocabulary more efficiently and generalize better to novel word combinations, mimicking human language acquisition.

What Problem Does This Paper Address?

Traditional vision-language models struggle with language acquisition for several key reasons: they require massive amounts of data to learn word embeddings, they generalize poorly to the natural compositional structure of language (e.g., combining known words in new ways), and they often fail to adapt to new, unseen words (out-of-vocabulary words). The authors aim to bridge this gap by developing a system that can learn to learn words, much like how children naturally acquire language by observing the world and inferring meaning from context.

Methods

The core of EXPERT is a meta-learning framework that trains a "meta-learner" to acquire word representations from visual scenes. Instead of directly learning word embeddings, the model learns a policy for acquiring them. This is achieved by presenting the model with "episodes" – sequences of image-text pairs. Within each episode, the model is tasked with reconstructing masked words by "pointing" to the correct word within reference examples in the same episode. This forces the model to learn a robust process for word acquisition.

The model architecture leverages self-attention-based transformers. It uses three complementary loss terms:

  • Pointing to Words: Encourages the model to correctly identify the reference word for a masked word.
  • Word Cloze: A standard language modeling objective to predict masked words directly.
  • Visual Cloze: A metric loss to improve visual representations by ensuring projected visual features are closer to the correct visual features than incorrect ones.

The authors also explore different attention mechanisms (e.g., isolated, target-to-reference, via-vision, full attention) to control information flow within episodes.

Datasets

The authors evaluate EXPERT on two datasets:

  • EPIC-Kitchens: A large dataset of egocentric video clips with narrated descriptions, featuring complex, unscripted actions.
  • Flickr30k: A dataset of images with descriptive captions, known for its diverse scenes and longer-tail word distribution.

For evaluation, they create train-test splits that withhold certain words and compositions from the training set, forcing the model to acquire them at test time.

Key Insights and Ideas

  • Learning to Learn is Key: The fundamental insight is that learning how to acquire words is more effective than directly learning word representations, especially for generalization and adaptation to new vocabulary.
  • Episodes as Training Ground: Using curated "episodes" of related image-text pairs allows the model to learn acquisition policies by observing how words relate to visual context and other words within a limited set of examples.
  • Visual Information is Crucial: Experiments show that visual information is vital for disambiguating new words and improving language acquisition, even when the model is trained without explicit object bounding boxes.
  • Data Efficiency: EXPERT achieves strong performance with significantly less training data compared to established baselines, even those pre-trained on massive text corpora.

Key Results

  • Novel Word Acquisition: EXPERT significantly outperforms baselines (BERT, BERT+Vision) in acquiring novel words, achieving up to 13% higher accuracy on EPIC-Kitchens and up to 37% higher accuracy on Flickr30k.
  • Compositional Generalization: The model shows a substantially smaller performance gap between seen and novel word compositions, indicating better generalization.
  • Data Efficiency: EXPERT matches or exceeds the performance of pretrained BERT on Flickr30k while using orders of magnitude less training data.
  • Robustness: Performance degrades less than baselines as the number of distractor examples in the reference set increases.
  • Visual Importance: Removing visual information leads to a significant drop in performance, highlighting the model's reliance on visual cues.

Limitations and Future Work

  • Error Cases: While generally strong, EXPERT can still make plausible but incorrect predictions, as shown in Figure 5 where it misinterprets "grabbed" as "placed" or "put." The authors suggest that video information could further improve performance.
  • Computational Cost: While some attention variants scale linearly, full attention can scale quadratically with episode size.
  • Future Work: The authors suggest that their meta-learning approach can be extended to other areas of language acquisition and that incorporating video information could further enhance performance. They also plan to release their code and trained models.

Summary updated

Abstract

Language acquisition is the process of learning words from the surrounding scene. We introduce a meta-learning framework that learns how to learn word representations from unconstrained scenes. We leverage the natural compositional structure of language to create training episodes that cause a meta-learner to learn strong policies for language acquisition. Experiments on two datasets show that our approach is able to more rapidly acquire novel words as well as more robustly generalize to unseen compositions, significantly outperforming established baselines. A key advantage of our approach is that it is data efficient, allowing representations to be learned from scratch without language pre-training. Visualizations and analysis suggest visual information helps our approach learn a rich cross-modal representation from minimal examples. Project webpage is available at https://expert.cs.columbia.edu/
BibTeX
@misc{suris2019learning,
  author = {Dídac Surís and Dave Epstein and Heng Ji and Shih-Fu Chang and Carl Vondrick},
  title = {{Learning to Learn Words from Narrated Video}},
  year = {2019},
  eprint = {1911.11237},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.11237},
}

Loading related content...

Figure