Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan L. Yuille

Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

Introduction

Modern image captioning systems typically rely on massive, static datasets and fixed vocabularies. When faced with a new object or activity, these models generally require full retraining on the entire combined dataset—a process that is computationally prohibitive and lacks the flexibility of human learning. This paper addresses the challenge of Novel Visual Concept learning from Sentences (NVCS), a framework designed to mimic the "fast mapping" ability seen in children, who can learn new words from just a few exposures in context.

The NVCS Challenge

The goal of NVCS is to enable a model to integrate new visual concepts (e.g., "quidditch" or "samisen") using only a handful of images accompanied by sentence descriptions. Unlike standard one-shot classification, this task requires the model to understand the linguistic role of the new word and its relationship with known concepts.

A comparison between a base model misidentifying quidditch as soccer and the NVCS model correctly identifying it after training on a few examples.

Methodology: Enhancing the m-RNN

The authors build upon the multimodal Recurrent Neural Network (m-RNN) architecture, introducing three key innovations to facilitate rapid and stable learning:

  1. Transposed Weight Sharing (TWS): To prevent overfitting on small novel datasets, the model shares weights between the word embedding layer ($U_D$) and the final SoftMax decoding layer. Specifically, the decoding layer uses the transpose of the embedding matrix ($U_D^T$). This reduces the parameter count and forces the model to learn more robust semantic representations.
  2. Incremental Weight Updates: To preserve existing knowledge, the model only updates the sub-matrix of weights corresponding to the new vocabulary ($U_{D_n}$), leaving the weights for previously learned words untouched.
  3. Baseline Probability Fixation (BPF): Small, concept-specific datasets often introduce bias, causing models to over-predict new words. BPF "centers" network activations to ensure new words are predicted only when appropriate, maintaining a balanced output distribution.
The m-RNN architecture (a) and the proposed Transposed Weight Sharing scheme (b) which halves parameters and improves stability.

Experimental Setup and Datasets

The framework was evaluated using the MS-COCO dataset as a base and a custom NC-3 (Novel Concept-3) dataset. NC-3 features three visually and linguistically unique concepts: Quidditch, T-Rex, and Samisen. These concepts were chosen because they are absent from standard benchmarks and appear in diverse sentence structures.

Key Results

  • Efficiency: The "Deep-NVCS" approach achieves performance comparable to full retraining while requiring less than 2% of the training time.
  • Sample Efficiency: The model demonstrates high sample efficiency, reaching a performance plateau with as few as 10 to 50 training images.
  • Semantic Integration: Analysis of the word-embedding space revealed that the model logically grouped new concepts with similar known ones (e.g., "quidditch" clustered with "soccer" and "football"), indicating a genuine understanding of semantic context.
  • Stability: By isolating updates to new vocabulary, the model avoids "catastrophic forgetting," maintaining its ability to describe original MS-COCO objects accurately.

Conclusion

This research provides a scalable path for image captioning models to evolve dynamically. By leveraging linguistic context and efficient weight-sharing strategies, the NVCS framework allows AI to learn new visual concepts with a speed and efficiency that approaches human-like learning. Future work will focus on handling style mismatches between datasets and scaling the system to accommodate thousands of incremental updates.

Summary updated

BibTeX
@inproceedings{mao2015learning,
  author = {Junhua Mao and Wei Xu and Yi Yang and Jiang Wang and Zhiheng Huang and Alan L. Yuille},
  title = {{Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=SyVC5gfdbS},
}
Figure