Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models

Anna Bavaresco, Alberto Testoni, Raquel FernándezACL 2024

TL;DR

This paper challenges the impressive zero-shot performance of contrastive vision-and-language models (VLMs) on ad understanding tasks, arguing that models exploit superficial text-image grounding rather than true comprehension. They introduce TRADE, a new adversarial benchmark, demonstrating that these VLMs perform at chance level on it, while humans excel, highlighting the need for more robust evaluation methods.

What Problem Does This Paper Address?

Automatic understanding of advertisements is a complex multimodal task, requiring models to grasp not only visual elements and text but also the nuanced persuasive messages conveyed through figurative language and unusual imagery. Previous research using the Pitt Ads dataset reported remarkably high zero-shot accuracy for VLMs on an ad-explanation retrieval task. However, this paper questions whether these models truly understand the ads or are simply leveraging simple heuristics. The core problem is that existing evaluation setups might be too easily solved by models that can identify superficial textual and visual correlations, rather than demonstrating genuine multimodal reasoning abilities.

Methods

The authors first conduct a quantitative analysis of the original Pitt Ads evaluation setup. They compute textual and visual grounding scores (e.g., text overlap, text similarity, object mention, caption similarity) and correlate them with VLM alignment scores. This analysis aims to uncover whether models exploit simple relationships between ad text and explanations, or between visual elements and explanations.

To address the identified flaws, they introduce TRADE (TRuly ADversarial ad understanding Evaluation). TRADE is a new diagnostic test set designed to control for the "grounding gap" between positive and negative explanations. It consists of ads with one plausible explanation and two adversarial negative explanations. These adversarial explanations are carefully crafted by human annotators to be textually and visually grounded in the ad (mentioning objects and text present) but clearly incompatible with the ad's message, making them implausible to humans. This setup aims to "fool" VLMs that rely on simple grounding heuristics.

Datasets

  • Pitt Ads dataset: Used in previous work for ad understanding tasks. It consists of image-based ads with explanations capturing their underlying message. The original task involved retrieving a plausible explanation from a set of candidates.
  • TRADE (TRuly ADversarial ad understanding Evaluation): A newly introduced benchmark dataset. It comprises 300 randomly selected ads from the Pitt Ads dataset, each paired with one positive explanation and two adversarial negative explanations.
  • TRADE-control: A version of TRADE where negative explanations are randomly sampled from other ads, serving as a baseline to compare against the adversarial negatives.

Key Insights and Ideas

  • Superficial Grounding is Key: VLMs, particularly contrastive ones, are strongly biased towards explanations that are textually and visually grounded in the ad, regardless of their actual plausibility.
  • Existing Benchmarks are Flawed: The original Pitt Ads evaluation setup allows models to exploit simple grounding heuristics, leading to inflated performance metrics that don't reflect true multimodal reasoning.
  • Adversarial Examples are Crucial: Carefully crafted adversarial examples, like those in TRADE, are necessary to expose the limitations of current VLMs and to truly assess their understanding capabilities.
  • Humans Excel Where Models Fail: Humans can easily distinguish plausible explanations from adversarial distractors, highlighting a significant gap between human and VLM performance on challenging ad understanding tasks.

Key Results

  • On the original Pitt Ads evaluation setup, VLMs like CLIP achieved excellent zero-shot accuracy (e.g., 95.2% reported by Jia et al., 2023).
  • However, on the new TRADE benchmark, the performance of tested VLMs (CLIP, ALIGN, LiT, ALBEF) dropped significantly, performing at chance level (around 33%). For instance, CLIP achieved only 34% accuracy on TRADE, compared to 98% on TRADE-control.
  • Human performance on TRADE was high, achieving 94% accuracy.
  • The analysis of grounding scores showed a positive correlation between grounding metrics and VLM alignment scores on the original setup, indicating that models were indeed leveraging these superficial cues. In TRADE, this gap between positive and negative explanations was significantly reduced.

Limitations and Future Work

  • Model Training: The study focuses on zero-shot performance. It remains an open question whether VLMs trained or fine-tuned on the Pitt Ads dataset would be more robust against adversarial explanations.
  • Task Formulation: The retrieval-based setup has limitations (e.g., inability to provide task-specific instructions) and might not be the most appropriate for evaluating ad understanding. Future research could explore generative tasks.
  • Grounding Metrics: The proposed grounding scores are indicators of general trends rather than precise metrics, as visual and textual alignment are complex constructs.
  • Ethical Considerations: The Pitt Ads dataset may contain offensive content or stereotypes. While efforts were made to mitigate this, full guarantees are difficult. The licensing of the images is also unclear.
  • Cultural Nuances: The study does not account for personal and cultural factors that influence ad perception. Future work should adopt evaluation protocols reflecting a diverse set of interpretations.

Summary updated

BibTeX
@inproceedings{bavaresco2024dont,
  author = {Anna Bavaresco and Alberto Testoni and Raquel Fernández},
  title = {{Don’t Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure