Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, Alan L. Yuille

Abstract

In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. We propose a transposed weight sharing scheme and methods to prevent overfitting. Experiments show that our method effectively learns novel visual concepts from a few examples without disturbing previously learned concepts.

Introduction

Traditional image captioning models are "closed-world" systems: they are trained on a fixed vocabulary and struggle to incorporate new concepts. Learning a new concept like "quidditch" typically requires adding thousands of images and retraining the entire network—a slow and expensive process. Furthermore, simple fine-tuning often leads to "catastrophic forgetting" of old concepts or "frequency bias," where the model over-predicts the new word.

The authors propose a framework for Novel Visual Concept learning from Sentences (NVCS), mimicking "fast mapping" in children—the ability to form a quick hypothesis about a new word's meaning based on visual cues and linguistic context.

The NVCS framework: A base model misidentifies quidditch as soccer, but after training on a few image-sentence pairs, the NVCS model correctly identifies the novel concept.

Methodology

The researchers built upon the m-RNN (multimodal Recurrent Neural Network) architecture, introducing two key technical innovations:

  • Transposed Weight Sharing (TWS): Standard models use separate weights for word embedding and SoftMax. TWS forces these layers to share transposed weights, cutting parameters in half and ensuring that a word's input and output meanings reside in the same semantic space.
  • Baseline Probability Fixation (BPF): To solve frequency bias, BPF centralizes activations and fixes the bias terms of new words to match the average of old words. This prevents the model from over-predicting new concepts just because they were the focus of recent training.

Datasets

The approach was evaluated on three datasets:

  1. NewObj-Cat & NewObj-Motor: Derived from MS-COCO by withholding "cat" or "motorcycle" images during base training.
  2. NC-3 (Novel Concept-3): A new benchmark featuring concepts absent from standard datasets: Quidditch, T-rex, and Samisen. Each concept has 100 images with 5 descriptive sentences each.
An example image from the NC-3 dataset showing a person playing quidditch, a concept not found in standard image datasets.

Key Results and Insights

  • Few-Shot Success: The model learns effectively with as few as 10 to 50 images. Training takes less than 2% of the time required for a full retrain.
  • Accuracy and Stability: On the NewObj-Cat dataset, the model achieved an F-score of 0.875, significantly outperforming baseline methods (0.183). Crucially, BPF allowed the model to learn new words without decreasing performance on original concepts.
  • Linguistic Scaffolding: The model uses surrounding words to hypothesize meanings. If it sees "A person is playing [NEW_WORD]," it correctly categorizes the word as a sport or instrument. Qualitative results show "quidditch" is semantically mapped near "soccer" and "frisbee."

Conclusion

The NVCS framework demonstrates that AI can rapidly expand its vocabulary by leveraging linguistic context and efficient weight sharing. While currently dependent on a strong pre-trained base, future work could move toward zero-shot learning, describing concepts purely from textual definitions.

Project page: www.stat.ucla.edu/junhua.mao/projects/child_learning.html

Summary updated

Abstract

In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. Our method has an image captioning module based on [38] with several improvements. In particular, we propose a transposed weight sharing scheme, which not only improves performance on image captioning, but also makes the model more suitable for the novel concept learning task. We propose methods to prevent overfitting the new concepts. In addition, three novel concept datasets are constructed for this new task, and are publicly available on the project page. In the experiments, we show that our method effectively learns novel visual concepts from a few examples without disturbing the previously learned concepts. The project page is: www.stat.ucla.edu/junhua. mao/projects/child_learning.html.
BibTeX
@inproceedings{mao2015learning,
  author = {Junhua Mao and Xu Wei and Yi Yang and Jiang Wang and Zhiheng Huang and Alan L. Yuille},
  title = {{Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=SyVC5gfdbS},
}
Figure