Image Captioning with Compositional Neural Module Networks

Junjiao Tian, Jean Oh

Introduction

Standard image captioning models often produce "safe" but generic descriptions, such as "a group of people," while ignoring fine-grained details. This "generality problem" stems from sequential encoder-decoder (RNN) approaches that prioritize next-word probability over specific visual attributes. To address this, we present a hierarchical framework using Compositional Neural Module Networks. This system selectively attends to specialized modules—including color, count, and size—to generate captions that are both linguistically fluent and rich in descriptive detail.

Architecture: The RNN Trio and Modular Detection

The core of the proposed method is a hierarchical framework that decomposes the captioning task into specialized sub-tasks. Unlike standard models that rely on a single LSTM, this architecture utilizes three distinct recurrent units:

  1. Attention (A)-LSTM: Determines which image regions are relevant at each time step.
  2. Visual (V)-LSTM: Generates an initial word prediction based purely on visual evidence.
  3. Semantic (S)-LSTM: Acts as the final decoder, synthesizing visual guesses and modular outputs into a coherent word.
The hierarchical framework featuring the RNN Trio and modular selection logic.

To identify objects, the model employs a Stacked Noisy-Or Object Detection strategy. This combines a Noisy-Or detector for individual regions with an Attention-based Multiple Instance Learning (MIL) detector for global context, allowing the system to capture both small objects and broad background elements like "grass" or "sky."

Compositional Reasoning and Attribute Modules

The system's innovation lies in its five specialized functional modules: Color, Count, Size, Spatial Relationship, and Semantic Relationship. At each time step, a Compositional Module Attention mechanism decides which module to query. For instance, when describing a "red apple," the gate prioritizes the Color module to ensure attribute accuracy.

Module attention visualization and the compositional pipeline.

This modularity provides significant visual interpretability. As shown in Figure 2, one can observe the model switching from an "initial estimation" to the "count" module when generating the word "two." This explicit delegation prevents the "averaging" effect common in standard LSTMs, where specific attributes are often lost in favor of high-frequency generic terms.

Results and Evaluation

The model was evaluated on the MSCOCO dataset using the Karpathy split. Results indicate that specialization improves semantic accuracy without sacrificing fluency. The proposed model outperformed state-of-the-art "Top-Down" attention models across several metrics:

  • CIDEr Score: Increased from 105.4 to 108.9.
  • SPICE (Semantic Metric): Improved from 19.9 to 20.4.
  • Attribute Specificity: Significant gains were observed in the Count (9.12 to 14.70) and Color (6.59 to 7.92) f-scores.

Human evaluations via Amazon Mechanical Turk further confirmed these findings, with a 14% higher preference for the modular model in images requiring specific counts.

Discussion and Limitations

While the modular approach excels at counting and color identification, the "Size" module showed limited improvement. Size is a relative concept requiring environmental comparison, which the current module—focusing primarily on the object and the whole image—struggles to capture. Future work may involve more sophisticated module architectures and improved grounding to handle occlusions and complex spatial relationships in cluttered scenes.

Summary updated

Abstract

In image captioning where fluency is an important factor in evaluation, e.g., $n$-gram metrics, sequential models are commonly used; however, sequential models generally result in overgeneralized expressions that lack the details that may be present in an input image. Inspired by the idea of the compositional neural module networks in the visual question answering task, we introduce a hierarchical framework for image captioning that explores both compositionality and sequentiality of natural language. Our algorithm learns to compose a detail-rich sentence by selectively attending to different modules corresponding to unique aspects of each object detected in an input image to include specific descriptions such as counts and color. In a set of experiments on the MSCOCO dataset, the proposed model outperforms a state-of-the art model across multiple evaluation metrics, more importantly, presenting visually interpretable results. Furthermore, the breakdown of subcategories $f$-scores of the SPICE metric and human evaluation on Amazon Mechanical Turk show that our compositional module networks effectively generate accurate and detailed captions.
BibTeX
@inproceedings{tian2019image,
  author = {Junjiao Tian and Jean Oh},
  title = {{Image Captioning with Compositional Neural Module Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/496},
}
Figure