A Neural Compositional Paradigm for Image Captioning

Bo Dai, Sanja Fidler, Dahua Lin

TL;DR

Standard sequential image captioners often hallucinate objects because they lean too heavily on frequent training n-grams rather than hierarchical sentence structure. This paper introduces CompCap, a two-stage paradigm that factorizes captioning into explicit semantic extraction (predicting noun phrases) and recursive bottom-up composition. By decoupling "what to say" from "how to say it," the model produces far more diverse and generalizable captions with fewer hallucinations.

What Problem Does This Paper Address?

Mainstream captioning architectures rely on an encoder-decoder framework: a convolutional neural network (CNN) encodes an image into a latent vector, and an RNN/LSTM generates words sequentially. Because natural language has an inherently hierarchical tree structure, sequential word-by-word generation creates several issues:

  1. Hallucinations driven by n-gram priors: If "building with a clock tower" is common in the training set, the model may append "with a clock tower" to any building, even if no clock exists.
  2. Entangled syntax and semantics: Fusing semantic content with grammatical structure makes models brittle, data-hungry, and prone to repetitive, cookie-cutter captions.
  3. Poor out-of-domain generalization: Sequential decoders struggle when tested on new distributions because their syntactic rules are glued to specific vocabulary co-occurrences.

Methods

The authors propose a two-stage framework that treats caption generation as the inverse of sentence parsing:

[ Image ] ──> [ Stage 1: Explicit Semantics ] ──> Pool of Noun Phrases
                                                        │
                      ┌─────────────────────────────────┘
                      ▼
        [ Stage 2: CompCap (Recursive Composition) ]
           ├─ Connecting Module (C-Module): Joins two phrases via a connector
           └─ Evaluation Module (E-Module): Tests if phrase is a complete caption
  1. Explicit Semantic Extraction: Rather than passing an opaque visual embedding directly to a text decoder, the model extracts a set of top-$n$ descriptive noun phrases (e.g., "a small dog", "a football") via multi-label classification over image features, followed by Semantic Non-Maximum Suppression (NMS) to eliminate redundancy.
  2. Recursive Composition (CompCap):
  3. Phrase Pool Maintenance: The framework initializes a phrase pool with the extracted noun phrases.
  4. Connecting Module (C-Module): Iterates over ordered phrase pairs $(P^{(l)}, P^{(r)})$ and selects a connecting phrase $P^{(m)}$ to yield $P^{(l)} \oplus P^{(m)} \oplus P^{(r)}$ (e.g., joining "a small dog" and "a football" with "playing with"). Because connector phrases rarely contain nouns or adjectives, the authors cast connector generation as a classification problem over ~1,000 mined functional sequences rather than sequence-to-sequence decoding. Phrase encodings are computed using a two-level attention LSTM.
  5. Evaluation Module (E-Module): Assesses whether the newly formed phrase constitutes a complete caption using an LSTM classifier. If complete, it stops; otherwise, it replaces the constituent phrases in the pool with the new composite phrase and repeats.

Datasets

The authors evaluate on two standard image-captioning benchmarks using the Karpathy splits:

  • MS-COCO: 123,287 images, each with 5 reference captions.
  • Flickr30k: 31,783 images, each with 5 reference captions.

No new datasets were introduced. Caption syntax trees for supervision were parsed using Stanford CoreNLP (NLPToolkit).

Key Insights and Ideas

  • Decoupling syntax from semantics: Syntax (sentence composition rules) is largely domain-agnostic, whereas semantics (objects and attributes) shifts between datasets. Separating them allows the syntactic composition module to generalize across datasets cleanly.
  • Treating connecting text as classification: Decoding variable-length connectors between incomplete sentence fragments via recurrent decoders typically fails. Formulating it as classifying over a closed vocabulary of ~1,000 structural connectors avoids this pitfall.
  • High SPICE vs. lower CIDEr/BLEU: CompCap scores higher on SPICE (which focuses on semantic propositional tuples) but lower on n-gram metrics like CIDEr and BLEU. This reflects that the model avoids parroting memorized, frequent training n-grams, producing valid captions that traditional overlap metrics penalize.

Key Results

  • Semantic Accuracy (SPICE): CompCap achieves the highest SPICE scores among compared methods on both MS-COCO (19.9 vs. 18.7 for TopDown) and Flickr30k (14.9 vs. 13.8 for TopDown).
  • Caption Diversity: On MS-COCO, CompCap dramatically increases novelty and diversity compared to TopDown:
  • Novel Caption Ratio: 90.48% vs. 45.05%
  • Unique Caption Ratio: 83.86% vs. 61.58%
  • Image-Level Diversity (edit distance): 5.57 vs. 2.30
  • Data Efficiency & Transfer: When trained on just 1% of COCO training data, CompCap maintains stable CIDEr and SPICE performance while baseline performance collapses. In cross-dataset transfer (train on COCO, test on Flickr30k, and vice versa), CompCap displays significantly smaller performance drops than sequential baselines.
  • Ablation Upper Bound: Providing ground-truth noun phrases skyrockets the SPICE score on COCO from 19.9 to 36.8, highlighting that the compositional engine effectively preserves provided semantic concepts.

Limitations and Future Work

  • Bottlenecked by Visual Concept Extraction: Because phrase composition is deterministic and modular, errors in the initial noun-phrase extraction step propagate directly to the final caption (e.g., misidentifying background elements as primary subjects).
  • Syntax Degradation with Larger Pools: Scaling beyond $n = 7$ noun phrases leads to a noticeable drop in CIDEr scores, indicating that attempting to integrate too many concepts risks syntactic coherence.
  • Penalty from Standard Overlap Metrics: Because it assembles phrases dynamically rather than mimicking training corpus statistics, it achieves lower BLEU-4 and CIDEr scores compared to models optimized directly for sequential likelihood or metric-based reinforcement learning.

Summary updated

Abstract

Mainstream captioning models often follow a sequential structure to generate cap- tions, leading to issues such as introduction of irrelevant semantics, lack of diversity in the generated captions, and inadequate generalization performance. In this paper, we present an alternative paradigm for image captioning, which factorizes the captioning procedure into two stages: (1) extracting an explicit semantic representation from the given image; and (2) constructing the caption based on a recursive compositional procedure in a bottom-up manner. Compared to conventional ones, our paradigm better preserves the semantic content through an explicit factorization of semantics and syntax. By using the compositional generation procedure, caption construction follows a recursive structure, which naturally fits the properties of human language. Moreover, the proposed compositional procedure requires less data to train, generalizes better, and yields more diverse captions.
BibTeX
@inproceedings{dai2018neural,
  author = {Bo Dai and Sanja Fidler and Dahua Lin},
  title = {{A Neural Compositional Paradigm for Image Captioning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=H14icd-dZH},
}

Loading related content...

Figure