Abstract: In image captioning where fluency is an important factor in evaluation, e.g., $n$-gram metrics, sequential models are commonly used; however, sequential models generally result in overgeneralized expressions that lack the details that may be present in an input image. Inspired by the idea of the compositional neural module networks in the visual question answering task, we introduce a hierarchical framework for image captioning that explores both compositionality and sequentiality of natural language. Our algorithm learns to compose a detail-rich sentence by selectively attending to different modules corresponding to unique aspects of each object detected in an input image to include specific descriptions such as counts and color. In a set of experiments on the MSCOCO dataset, the proposed model outperforms a state-of-the art model across multiple evaluation metrics, more importantly, presenting visually interpretable results. Furthermore, the breakdown of subcategories $f$-scores of the SPICE metric and human evaluation on Amazon Mechanical Turk show that our compositional module networks effectively generate accurate and detailed captions.
1. Introduction
Standard image captioning models typically follow a sequential encoder-decoder architecture. While these models are proficient at producing fluent, natural-sounding sentences, they tend to suffer from "generic-itis"—a tendency to produce safe but overgeneralized descriptions. For example, a model might describe an image as containing "a group of people" rather than the more precise "three children." This occurs because models are trained to maximize likelihood on datasets where generic labels are statistically more common than specific ones.
The core challenge lies in the hierarchical nature of natural language: objects possess attributes grounded in specific visual regions. Sequential models often overlook this structure, leading to descriptions that lack the specificity required for advanced AI assistants or tools for the visually impaired. This paper introduces a hierarchical framework using Compositional Neural Module Networks to bridge the gap between fluency and visual detail.
2. Methods
The proposed model combines the fluency of sequential generation with the precision of specialized modular "experts." The architecture is structured around three primary components:
2.1 The RNN Trio
The framework replaces the standard single LSTM with a trio of specialized units:
- Attention-LSTM: Responsible for determining which image region to focus on at each time step.
- Visual-LSTM: Provides an initial captioning guess based purely on visual evidence from the attended region.
- Semantic-LSTM: Functions as the final "composer," integrating object and attribute detections to predict the next word in the sequence.
2.2 Stacked Noisy-Or Object Detection
To enhance object detection, the model employs a "stacked" approach. It integrates a region-based detector, which is effective for small, distinct objects, with an attention-based detector that better captures large background elements such as "sky" or "grass."
2.3 Modular Attribute Detection
The defining feature of this framework is the inclusion of five specialized modules: Color, Count, Size, Spatial, and Semantic. A Module Attention mechanism dynamically selects which module to query based on the current context of the sentence being generated. For instance, if the model is about to generate a numeral, the "Count" module's influence is maximized.
3. Architecture and Interpretability
The model learns a specific "layout" for each sentence, deciding when to invoke the "Color" expert versus the "Spatial" expert. Unlike global models, this framework ties attributes directly to proposed object regions. This grounding ensures that attributes like "red" are associated with a specific "apple" at defined coordinates rather than being loosely associated with the entire scene.
Furthermore, this modularity provides significant interpretability. By visualizing the "Module Attention," researchers can verify the reasoning behind specific word choices. If the model outputs "two," one can confirm that the "Count" module was the primary driver of that decision.
4. Experimental Results
The model was trained on the MSCOCO dataset, with a Faster-RCNN backbone pre-trained on Visual Genome for region proposals. It outperformed the state-of-the-art "Top-Down" baseline across all major metrics on the MSCOCO Karpathy test split:
- CIDEr: 108.9 (vs. 105.4)
- SPICE: 20.4 (vs. 19.9)
- BLEU-4: 33.0 (vs. 32.0)
The most significant improvements were observed in the SPICE subcategories, where the "Count" f-score rose from 9.12 to 14.70, and "Color" increased from 6.59 to 7.92. Human evaluations on Amazon Mechanical Turk corroborated these results, with users showing a 14% preference lead for captions generated by the modular model in images containing specific counts.
5. Conclusion and Future Work
While the compositional approach significantly improves the detection of counts and colors, relative attributes like "Size" remain a challenge. The current two-layer MLP design for the size module is likely insufficient for capturing relative scale, which requires environmental comparison. Additionally, the model occasionally experiences "attribute leakage," where a color is incorrectly associated with a nearby object. Future research will focus on more sophisticated module architectures and enhanced spatial reasoning to address these limitations.
Summary updated