KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

Zhiwei Jia, Pradyumna Narayana, Arjun R. Akula, Garima Pruthi, Hao Su, Sugato Basu, Varun Jampani

TL;DR

This paper tackles the challenge of understanding image advertisements, a complex task that goes beyond standard image recognition. The authors propose KAFA, a novel approach that enhances existing Vision-Language Models (VLMs) by incorporating external knowledge about brands and products, leading to significantly improved ad comprehension.

What Problem Does This Paper Address?

Image advertisements are a crucial part of our society, driving applications like ad targeting, visual metaphor analysis, and creative ad generation. However, understanding them is surprisingly difficult. Ads often feature diverse, non-photorealistic elements, atypical scenes, and require knowledge of a vast number of real-world entities like brands and products. Existing methods struggle with this complexity, especially in the era of powerful foundational VLMs. While VLMs like CLIP, ALBEF, and LiT show impressive generalizability, their direct application to image ad understanding reveals practical challenges, such as overfitting and the computational burden of effective training. This paper aims to bridge this gap by proposing a method to adapt VLMs for this specific task, leveraging external knowledge to overcome these limitations.

Methods

The core of the proposed solution, KAFA (Knowledge-Augmented Feature Adaptation), involves two main components:

  • Attention-based Feature Adaptation: This is a lightweight neural module that fuses information from different modalities (image, scene-text, and brand knowledge). It uses a multi-head attention mechanism to combine these features into a unified representation. This approach is designed to be efficient and to avoid overfitting, a common issue when fine-tuning VLMs on limited ad data.
  • Brand Understanding Module: To inject crucial real-world knowledge, the authors construct a large knowledge base called "BrandSet-110K," containing around 110,000 brand names and their descriptions. This module uses an ensemble of text-matching (leveraging OCR-extracted scene text) and vision-based recognition (using a VLM to propose regions and CLIP for logo recognition) to identify brands within an ad. This module is training-free and aims to provide comprehensive brand coverage, outperforming existing methods and commercial APIs.

KAFA integrates these components by feeding the adapted features into a VLM for downstream tasks.

Datasets

  • Pitt Dataset: This dataset, containing image ads with associated text descriptions answering "what should I do according to the ad and why?", is used for both training and evaluation. The paper notes that it contains 51,223 image-text pairs.
  • BrandSet-110K: A newly constructed knowledge base of approximately 110,000 brand names and their descriptions, used by the brand understanding module.

Key Insights and Ideas

  • Pre-training Scale Matters Most: The sheer scale of data and model capacity used during VLM pre-training is the most critical factor for image ad understanding performance. This is partly because VLMs implicitly store a lot of real-world knowledge.
  • Standard Metrics Fall Short: Common metrics for evaluating VLMs (like ImageNet accuracy or Flickr30K retrieval) do not accurately reflect their performance on image ad understanding tasks. This suggests that image ad understanding requires a deeper grasp of real-world entities and context.
  • Overfitting is a Major Hurdle: Adapting VLMs to image ads is prone to overfitting due to limited training data and weak supervision signals.
  • Hard Negative Mining is Computationally Expensive: While effective for training, fully online hard negative mining (HNM) is computationally intensive, posing a practical challenge.
  • External Brand Knowledge is Crucial: Explicitly incorporating knowledge about brands significantly improves image ad understanding, highlighting the importance of real-world entity recognition.

Key Results

  • KAFA Outperforms Baselines: The proposed KAFA approach, with its attention-based feature adapter and brand understanding module, achieves substantial improvements in image ad understanding over VLM baselines.
  • Zero-Shot Performance: The best CLIP model, even in a zero-shot setting, surpasses previous state-of-the-art results on the Pitt Dataset, demonstrating the power of pre-trained VLMs.
  • Ablation Studies: Experiments show that KAFA with both scene-text and brand knowledge input performs best. Using only image features with the adapter (KAFA w/o K) still outperforms CLIP-Adapter, and adding scene-texts (KAFA w/o ST) further boosts performance.
  • Brand Recognition Accuracy: The brand understanding module achieves competitive brand recognition accuracy on a validation set, justifying its importance for ad understanding.

Limitations and Future Work

  • Data Bias: The Pitt Dataset, while useful, may contain images and annotations that perpetuate harmful stereotypes, and the authors plan to collaborate with AI robustness researchers to address this.
  • Future Directions: The authors suggest extending their work to directly generating text descriptions for image ads or even generating image ads from descriptions. They also believe their study can provide insights for the broader machine learning community.

Summary updated

BibTeX
@inproceedings{jia2023kafa,
  author = {Zhiwei Jia and Pradyumna Narayana and Arjun R. Akula and Garima Pruthi and Hao Su and Sugato Basu and Varun Jampani},
  title = {{KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=MysWDqLQKhE},
}
Figure