Abstract
Standard image captioning models struggle to describe objects not present in their training captions, a challenge known as Novel Object Captioning (NOC). This paper introduces Feature Deformation Meta-Networks (FDM-net), a framework that addresses this by synthesizing training data in the latent feature space. By swapping Region of Interest (RoI) features of known objects with those of novel ones and using scene graphs to maintain linguistic consistency, FDM-net enables models to describe unseen objects without requiring new paired image-sentence data.
Introduction: The Challenge of Novel Objects
Image captioning has seen significant progress, yet models remain notoriously "data-hungry." They typically fail when encountering novel objects—items that may appear in object detection datasets but were absent from the specific image-caption pairs used for training. For instance, a model trained on "dog on a chair" might fail to generate "dog on a couch" if "couch" was never described in its training set, even if it can detect a couch.
The primary bottleneck is the lack of paired training data. While raw image editing (e.g., copy-pasting a couch into a scene) can create synthetic data, it often introduces visual artifacts like harsh edges or inconsistent lighting, which can confuse the learning process.
The FDM-net Framework
FDM-net bypasses the issues of pixel-level editing by operating directly on high-level visual features. The framework consists of two primary components designed to create high-quality synthetic training instances:
1. Feature Deformation Sub-net
Instead of manipulating raw pixels, FDM-net works with Region of Interest (RoI) features extracted by a pre-trained detector (such as Faster R-CNN).
- Mis-labelled Probability Strategy (MLS): The model identifies "seen" objects that are visually similar to "novel" ones (e.g., a chair is similar to a couch).
- Feature Swapping: It takes the feature map of a known scene, removes the feature vector of the "seen" object, and replaces it with the feature vector of a novel object. This "latent augmentation" avoids the noise associated with image-level manipulation.
2. Scene Graph Sentence Reconstruction (SGSR)
To ensure the generated caption remains accurate after feature swapping, FDM-net employs scene graphs as a structured intermediary.
- Graph Parsing: A training sentence (e.g., "A dog sits on a chair") is parsed into a scene graph.
- Node Replacement: The "chair" node is replaced with "couch."
- Attribute Update: An attribute classifier updates relevant descriptors to ensure the new sentence is semantically sound.
- Decoding: The modified graph is decoded back into a coherent, natural language sentence.
Experimental Results
The authors evaluated FDM-net on the MSCOCO Novel Object Captioning (NOC) split, excluding eight specific classes (e.g., bottle, couch, zebra) from training captions. Visual features for these novel objects were sourced from Open Images V4, while Visual Genome provided the linguistic grounding for the SGSR sub-net.
Key Performance Metrics
FDM-net achieved state-of-the-art results, significantly outperforming previous baselines like DCC and NBT:
- SPICE Score: 19.6
- METEOR Score: 25.6
- F1 Score Boost: The model's ability to correctly mention novel objects improved from 0% (baseline) to 64.7%.
Human evaluations further validated these results, with participants preferring FDM-net's captions nearly 50% of the time over the previous state-of-the-art, noting superior object incorporation and descriptive detail.
Conclusion and Future Work
FDM-net demonstrates that augmenting data in the latent feature space is a powerful alternative to raw image manipulation for novel object captioning. By leveraging visual similarities and structured scene graphs, the model bridges the gap between object detection and natural language generation.
Future research may focus on improving the fluency of reconstructed sentences for complex scene graphs and reducing the system's dependency on the initial quality of the object detector's RoI features.
Summary updated