Abstract
In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. We propose a transposed weight sharing scheme and methods to prevent overfitting. Experiments show that our method effectively learns novel visual concepts from a few examples without disturbing previously learned concepts.
Introduction
Traditional image captioning models are "closed-world" systems: they are trained on a fixed vocabulary and struggle to incorporate new concepts. Learning a new concept like "quidditch" typically requires adding thousands of images and retraining the entire network—a slow and expensive process. Furthermore, simple fine-tuning often leads to "catastrophic forgetting" of old concepts or "frequency bias," where the model over-predicts the new word.
The authors propose a framework for Novel Visual Concept learning from Sentences (NVCS), mimicking "fast mapping" in children—the ability to form a quick hypothesis about a new word's meaning based on visual cues and linguistic context.
Methodology
The researchers built upon the m-RNN (multimodal Recurrent Neural Network) architecture, introducing two key technical innovations:
- Transposed Weight Sharing (TWS): Standard models use separate weights for word embedding and SoftMax. TWS forces these layers to share transposed weights, cutting parameters in half and ensuring that a word's input and output meanings reside in the same semantic space.
- Baseline Probability Fixation (BPF): To solve frequency bias, BPF centralizes activations and fixes the bias terms of new words to match the average of old words. This prevents the model from over-predicting new concepts just because they were the focus of recent training.
Datasets
The approach was evaluated on three datasets:
- NewObj-Cat & NewObj-Motor: Derived from MS-COCO by withholding "cat" or "motorcycle" images during base training.
- NC-3 (Novel Concept-3): A new benchmark featuring concepts absent from standard datasets: Quidditch, T-rex, and Samisen. Each concept has 100 images with 5 descriptive sentences each.
Key Results and Insights
- Few-Shot Success: The model learns effectively with as few as 10 to 50 images. Training takes less than 2% of the time required for a full retrain.
- Accuracy and Stability: On the NewObj-Cat dataset, the model achieved an F-score of 0.875, significantly outperforming baseline methods (0.183). Crucially, BPF allowed the model to learn new words without decreasing performance on original concepts.
- Linguistic Scaffolding: The model uses surrounding words to hypothesize meanings. If it sees "A person is playing [NEW_WORD]," it correctly categorizes the word as a sport or instrument. Qualitative results show "quidditch" is semantically mapped near "soccer" and "frisbee."
Conclusion
The NVCS framework demonstrates that AI can rapidly expand its vocabulary by leveraging linguistic context and efficient weight sharing. While currently dependent on a strong pre-trained base, future work could move toward zero-shot learning, describing concepts purely from textual definitions.
Project page: www.stat.ucla.edu/junhua.mao/projects/child_learning.html
Summary updated