Understanding Advertisements with BERT

Kanika Kalra, Bhargav Kurma, Silpa Vadakkeeveetil Sreelatha, Manasi S. Patwardhan, Shirish Subhash KarandeACL 2020 · OpenReview

Abstract: Kanika Kalra, Bhargav Kurma, Silpa Vadakkeeveetil Sreelatha, Manasi Patwardhan, Shirish Karande. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

2020.

Introduction

Understanding an advertisement is significantly more complex than standard object detection. While a typical computer vision model might identify "red liquid" in a "glass," a human viewer interprets these elements as symbols for "danger" or "refreshment" depending on the creative context. Ad understanding requires a deep interpretation of symbolism, intent, and the nuanced relationship between slogans and visual cues.

The core challenge addressed in this research is the Action-Reason Pair (ARP) task. Given an advertisement image, the model must correctly identify why a viewer should take a specific action (e.g., "I should buy this car because it adds excitement to my life"). Previous state-of-the-art attempts relied on complex, multi-branch neural networks designed to fuse image pixels with text. This paper explores a provocative alternative: skipping complex pixel-level fusion entirely and instead describing the image to a powerful language model.

Methodology: A Text-Only Pipeline

The researchers proposed a pipeline that treats advertisement understanding as a pure natural language processing task. The approach involves three primary stages:

  1. Visual-to-Text Conversion:
    • Scene-Text (ST): The Google Vision API is utilized to extract embedded text such as slogans and brand names.
  2. Dense Captions (C): A pre-trained Dense Captioning model generates descriptive sentences about objects, their properties, and their interactions (e.g., "a woman practicing yoga on a beach").
  3. Input Formatting: The extracted scene-text and dense captions are concatenated to form a "composite textual signal" that represents the visual content of the ad.
  4. BERT Fine-Tuning: The task is framed as a sentence-pair classification problem. Sentence A consists of the composite textual signal (ST + C), while Sentence B is the candidate viewer interpretation (the ARP). The model then performs a binary classification to determine if the interpretation is valid or invalid.
The proposed pipeline converts visual elements into text (Scene-Text and Dense Captions) to be processed by a BERT model for Action-Reason Pair (ARP) classification.

By leveraging BERT’s self-attention mechanism, the model learns to associate specific words in the interpretation (like "empowering") with specific cues in the image description or slogan (like "Power of the she").

Experimental Results

The study utilized the CVPR 2018 Advertisement Understanding dataset, which contains over 64,000 images. Each image is paired with 3 to 5 valid ARPs provided by human annotators and several negatively sampled invalid ARPs.

The proposed model, BERT FT ST+C, established a new benchmark for the dataset, achieving an accuracy of 89.69% on the test set. This represents a significant leap over previous multi-modal architectures.

Comparison of the BERT-based approach against various multi-modal baselines on the CVPR 2018 Ads dataset.

As shown in the results table, the BERT-based approach significantly outperformed specialized cross-modal transformers like ViLBERT (61.76%) and LXMERT (50.00%), as well as the previous state-of-the-art model, ADVISE (69%).

Key Insights and Analysis

  • The Dominance of Scene-Text: The authors found that text embedded within the ad is the strongest signal for meaning. Using BERT with only scene-text achieved 84.95% accuracy, outperforming many complex visual models.
  • Abstraction via Text: Converting images into "concept-level" text allows the model to leverage BERT’s massive pre-existing knowledge of language and logic. This abstraction helps bridge the gap between raw pixels and high-level semantic reasoning.
  • Ablation Study: Using only Dense Captions (without scene-text) resulted in a lower accuracy of 60.09%. This confirms that while visual descriptions provide context, the ad's slogan is often the essential "key" to its intended meaning.

Limitations and Future Work

Despite its success, the system faces challenges. It is highly dependent on the quality of the scene-text extractor; the authors noted that OCR often misses stylized fonts or low-resolution text. Additionally, current dense captioning models can produce "spurious" or noisy captions. Future work may focus on capturing more granular details, such as specific emotions or complex object interactions, to further refine the model's understanding.

In conclusion, this research demonstrates that for tasks requiring high-level semantic reasoning, "describing" the world to a language model can be more effective than teaching a vision model to interpret abstract concepts directly from pixels.

Summary updated

BibTeX
@inproceedings{kalra2020understanding,
  author = {Kanika Kalra and Bhargav Kurma and Silpa Vadakkeeveetil Sreelatha and Manasi S. Patwardhan and Shirish Subhash Karande},
  title = {{Understanding Advertisements with BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=ax6XZjzFblI},
}
Figure