Towards Multimodal Vision-Language Models Generating Non-generic Text

Wes RobbinsAAAI 2022 · DOI

Abstract

Standard image captioning models often produce generic descriptions, failing to identify specific entities like famous people or unique text. This paper introduces the M4C + Special Tokens (M4C+ST) model, which integrates auxiliary classifiers into a multimodal transformer framework. By utilizing a "special token" modality and a dynamic pointer network, the model can generate specific, informative captions that include names and OCR-derived text. Evaluated on the new PAC dataset, M4C+ST demonstrates significant improvements in caption specificity and accuracy.

The Challenge of Specificity

Current image captioning models are technically proficient but often lack the "shared knowledge" necessary for socially relevant descriptions. A standard model might describe a world leader as "a man in a suit," which, while accurate, misses the most informative part of the image. While Optical Character Recognition (OCR) has helped models "read" text, identifying specific people remains a challenge. Most architectures are not designed to ingest labels from external specialized classifiers—such as face recognizers—and integrate them grammatically into sentences.

Example of an image processed with auxiliary classifiers (OCR and face recognition) to identify specific entities like Bernie Sanders and on-screen text.

Methods: The M4C+ST Architecture

The authors propose the M4C + Special Tokens (M4C+ST) model, extending the Multimodal Multi-Copy (M4C) framework. The core innovation is the integration of a "Special Token" modality alongside traditional visual features.

The M4C+ST architecture integrates special token embeddings and object embeddings into a multimodal transformer, using a dynamic pointer network for caption generation.

The process involves three main stages:

  1. Auxiliary Classification: Upstream tools, such as Google Cloud OCR and facial recognition (e.g., ArcFace), identify text and famous individuals.
  2. Multimodal Transformer Input: The model processes three types of embeddings:
  3. Generic Object Embeddings: Visual features of common objects (e.g., "man," "podium").
  4. Special Token Embeddings: Placeholders for specific names or text identified by auxiliary tools.
  5. Previous Prediction Embeddings: To maintain linguistic coherence during generation.
  6. Dynamic Pointer Network: Instead of relying solely on a fixed vocabulary, the model employs a copy mechanism. It can "point" to a special token (like "Bernie Sanders") and insert it directly into the caption, enabling the use of words outside its original training set.

Datasets and Evaluation

To evaluate the model's ability to generate non-generic text, the authors introduced the PAC (Politicians and Athletes in Captions) dataset. PAC contains 1,572 images of well-known individuals with captions that emphasize the integration of names and OCR tokens. The model was also trained on TextCaps, a large-scale dataset focused on reasoning about text in images.

Key Results

The M4C+ST model achieved substantial quantitative gains over the baseline M4C model on the PAC dataset. Notably, the CIDEr score jumped from 4.3 to 18.7 (a ~334% increase), and the BLEU-4 score rose from 2.1 to 9.1.

Qualitatively, the model successfully transitioned from ignoring famous individuals to identifying them and their context. For instance, where a baseline might say "a man standing," M4C+ST correctly identifies "Bernie Sanders standing in front of a screen that says 'this week'."

Discussion and Future Work

The "Special Token" approach treats names as specific descriptors for generic objects, allowing for zero-shot potential where the model can include names it hasn't seen in training if the auxiliary classifier identifies them. However, the model's accuracy is tethered to the reliability of these upstream classifiers. Future work aims to expand the dataset scale beyond politicians and athletes and integrate broader open-domain knowledge bases to provide even deeper contextual understanding.

Summary updated

BibTeX
@inproceedings{robbins2022towards,
  author = {Wes Robbins},
  title = {{Towards Multimodal Vision-Language Models Generating Non-generic Text}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21705},
}
Figure