Feature Deformation Meta-Networks in Image Captioning of Novel Objects

Tingjia Cao, Ke Han, Xiaomei Wang, Lin Ma, Yanwei Fu, Yu-Gang Jiang, Xiangyang Xue

Abstract

Standard image captioning models struggle to describe objects not present in their training captions, a challenge known as Novel Object Captioning (NOC). This paper introduces Feature Deformation Meta-Networks (FDM-net), a framework that addresses this by synthesizing training data in the latent feature space. By swapping Region of Interest (RoI) features of known objects with those of novel ones and using scene graphs to maintain linguistic consistency, FDM-net enables models to describe unseen objects without requiring new paired image-sentence data.

Introduction: The Challenge of Novel Objects

Image captioning has seen significant progress, yet models remain notoriously "data-hungry." They typically fail when encountering novel objects—items that may appear in object detection datasets but were absent from the specific image-caption pairs used for training. For instance, a model trained on "dog on a chair" might fail to generate "dog on a couch" if "couch" was never described in its training set, even if it can detect a couch.

The primary bottleneck is the lack of paired training data. While raw image editing (e.g., copy-pasting a couch into a scene) can create synthetic data, it often introduces visual artifacts like harsh edges or inconsistent lighting, which can confuse the learning process.

The FDM-net Framework

FDM-net bypasses the issues of pixel-level editing by operating directly on high-level visual features. The framework consists of two primary components designed to create high-quality synthetic training instances:

1. Feature Deformation Sub-net

Instead of manipulating raw pixels, FDM-net works with Region of Interest (RoI) features extracted by a pre-trained detector (such as Faster R-CNN).

  • Mis-labelled Probability Strategy (MLS): The model identifies "seen" objects that are visually similar to "novel" ones (e.g., a chair is similar to a couch).
  • Feature Swapping: It takes the feature map of a known scene, removes the feature vector of the "seen" object, and replaces it with the feature vector of a novel object. This "latent augmentation" avoids the noise associated with image-level manipulation.

2. Scene Graph Sentence Reconstruction (SGSR)

To ensure the generated caption remains accurate after feature swapping, FDM-net employs scene graphs as a structured intermediary.

  • Graph Parsing: A training sentence (e.g., "A dog sits on a chair") is parsed into a scene graph.
  • Node Replacement: The "chair" node is replaced with "couch."
  • Attribute Update: An attribute classifier updates relevant descriptors to ensure the new sentence is semantically sound.
  • Decoding: The modified graph is decoded back into a coherent, natural language sentence.
The FDM-net process: swapping 'chair' features for 'couch' features and updating the caption accordingly.

Experimental Results

The authors evaluated FDM-net on the MSCOCO Novel Object Captioning (NOC) split, excluding eight specific classes (e.g., bottle, couch, zebra) from training captions. Visual features for these novel objects were sourced from Open Images V4, while Visual Genome provided the linguistic grounding for the SGSR sub-net.

Key Performance Metrics

FDM-net achieved state-of-the-art results, significantly outperforming previous baselines like DCC and NBT:

  • SPICE Score: 19.6
  • METEOR Score: 25.6
  • F1 Score Boost: The model's ability to correctly mention novel objects improved from 0% (baseline) to 64.7%.

Human evaluations further validated these results, with participants preferring FDM-net's captions nearly 50% of the time over the previous state-of-the-art, noting superior object incorporation and descriptive detail.

Conclusion and Future Work

FDM-net demonstrates that augmenting data in the latent feature space is a powerful alternative to raw image manipulation for novel object captioning. By leveraging visual similarities and structured scene graphs, the model bridges the gap between object detection and natural language generation.

Future research may focus on improving the fluency of reconstructed sentences for complex scene graphs and reducing the system's dependency on the initial quality of the object detector's RoI features.

Summary updated

Abstract

This paper studies the task of image captioning with novel objects, which only exist in testing images. Intrinsically, this task can reflect the generalization ability of models in understanding and captioning the semantic meanings of visual concepts and objects unseen in training set, sharing the similarity to one/zero-shot learning. The critical difficulty thus comes from that no paired images and sentences of the novel objects can be used to help train the captioning model. Inspired by recent work (Chen et al. 2019b) that boosts one-shot learning by learning to generate various image deformations, we propose learning meta-networks for deforming features for novel object captioning. To this end, we introduce the feature deformation meta-networks (FDM-net), which is trained on source data, and learn to adapt to the novel object features detected by the auxiliary detection model. FDM-net includes two sub-nets: feature deformation, and scene graph sentence reconstruction, which produce the augmented image features and corresponding sentences, respectively. Thus, rather than directly deforming images, FDM-net can efficiently and dynamically enlarge the paired images and texts by learning to deform image features. Extensive experiments are conducted on the widely used novel object captioning dataset, and the results show the effectiveness of our FDM-net. Ablation study and qualitative visualization further give insights of our model.
BibTeX
@inproceedings{cao2020feature,
  author = {Tingjia Cao and Ke Han and Xiaomei Wang and Lin Ma and Yanwei Fu and Yu-Gang Jiang and Xiangyang Xue},
  title = {{Feature Deformation Meta-Networks in Image Captioning of Novel Objects}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6620},
}
Figure