KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

Zhiwei Jia, Pradyumna Narayana, Arjun R. Akula, Garima Pruthi, Hao Su, Sugato Basu, Varun Jampani

Abstract

Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.

Introduction

Understanding an image advertisement is significantly more complex than standard image captioning. While a typical computer vision model might identify "a bottle of soda on a beach," a human viewer perceives the underlying message of "refreshment and summer fun." Advertisements frequently employ visual metaphors, non-photorealistic elements, and creative scene-text to convey persuasive messages.

Despite the success of foundational Vision-Language Models (VLMs) like CLIP, interpreting the rhetoric of ads remains a significant hurdle. This paper introduces KAFA (Knowledge-Augmented Feature Adaptation), a framework designed to bridge the gap between raw visual recognition and high-level semantic understanding in the advertising domain.

The Challenges of Ad Understanding

The researchers identify three primary obstacles that hinder current AI models in this space:

  1. Atypical Scenes: Ads often use creative synthesis and surreal imagery that do not appear in standard academic datasets.
  2. Entity Knowledge: Interpreting an ad often requires specific knowledge of a brand's industry (e.g., recognizing that a logo represents a cleaning product rather than a food item).
  3. Multimodal Rhetoric: The core message is frequently hidden in the interplay between the visual elements and the "scene-text" (text embedded within the image).

Existing VLMs possess broad general knowledge but often struggle with these fine-grained requirements. Furthermore, directly fine-tuning these large models on specialized, smaller advertising datasets often leads to overfitting.

The KAFA Framework

To address these challenges, the KAFA framework utilizes a multi-pronged approach to feature adaptation:

1. Frozen VLM Backbone

KAFA utilizes powerful, pre-trained models such as CLIP ViT-L/14. By keeping the weights of the backbone "frozen," the framework prevents the catastrophic forgetting and overfitting that typically occurs during standard fine-tuning on small datasets, while also reducing computational overhead.

2. Brand Understanding Module

This training-free pipeline identifies brands within an image using an ensemble of OCR-based text matching and a vision-based module. It utilizes MAVL for region proposals and CLIP for classification to query a massive knowledge base, providing the model with the necessary context regarding the entities involved.

3. Attention-based Feature Adapter

Instead of traditional fine-tuning, KAFA employs a lightweight adapter layer. This layer uses multi-head attention to fuse features from three distinct sources: the original image, scene-text extracted via OCR, and the identified brand information. This allows the model to reason across different modalities simultaneously.

4. Hard Negative Mining (HNM)

To improve robustness, the model is trained using "hard negatives"—incorrect answers that are semantically similar to the correct interpretation. This forces the model to learn subtle distinctions between similar visual and textual cues.

Datasets and Benchmarking

The research utilizes two primary data resources:

  • Pitt Image Ads Dataset: The primary benchmark, containing approximately 51,000 image-text pairs where the text explains the intended action and rationale of the ad.
  • BrandSet-110K: A new knowledge base introduced by the authors. It contains roughly 110,000 entries of brands and organizations, each paired with a one-sentence description to provide the model with real-world industry context.

Key Results and Insights

KAFA achieved a new state-of-the-art accuracy of 97.4% on the Pitt Dataset, a significant improvement over the previous best of 89.7%.

Beyond raw accuracy, the study revealed several key insights:

  • The Importance of Scale: The scale of pre-training data is the single most influential factor in VLM performance for ads. CLIP (trained on 400M pairs) significantly outperformed models like ALBEF (trained on 14M pairs).
  • The "Brand" Gap: Many ads are incomprehensible without knowing the brand's industry. Explicitly feeding brand descriptions into the model provided the "missing link" for effective reasoning.
  • Adapters vs. Fine-tuning: The researchers found that directly fine-tuning a VLM on the Pitt dataset actually decreased performance. The lightweight adapter approach proved both more efficient and more accurate.
  • Robustness in Evaluation: In a "hard" evaluation setting—where the model must choose from 1,000 candidate answers rather than 15—KAFA maintained a robust 66.0% accuracy, while the zero-shot CLIP baseline dropped to 56.5%.

Conclusion and Future Work

While KAFA represents a significant step forward in ad retrieval and understanding, the authors note that challenges remain. Some ads are so abstract that they elude even human interpretation, and the Pitt dataset itself contains biases that reflect real-world stereotypes. Future work will likely shift from retrieval-based tasks to generative understanding, where models must synthesize and write their own explanations for complex visual metaphors from scratch.

Summary updated

Abstract

Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.
BibTeX
@inproceedings{jia2023kafa,
  author = {Zhiwei Jia and Pradyumna Narayana and Arjun R. Akula and Garima Pruthi and Hao Su and Sugato Basu and Varun Jampani},
  title = {{KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=MysWDqLQKhE},
}
Figure