Abstract
There is more to images than their objective physical content: for example, advertisements are created to persuade a viewer to take a certain action. We propose the novel problem of automatic advertisement understanding. To enable research on this problem, we create two datasets: an image dataset of 64,832 image ads, and a video dataset of 3,477 ads. Our data contains rich annotations encompassing the topic and sentiment of the ads, questions and answers describing what actions the viewer is prompted to take and the reasoning that the ad presents to persuade the viewer ("What should I do according to this ad, and why should I do it?"), and symbolic references ads make (e.g. a dove symbolizes peace). We also analyze the most common persuasive strategies ads use, and the capabilities that computer vision systems should have to understand these strategies. We present baseline classification results for several prediction tasks, including automatically answering questions about the messages of the ads.
Introduction: Beyond Literal Recognition
Most computer vision research focuses on objective physical content—identifying people, cars, or trees. However, advertisements are designed with intent; they utilize visual rhetoric to persuade. For example, an ad showing trees shaped like lungs is not a study in forestry; it is a symbolic message regarding environmental health and the act of breathing.
Current AI models excel at literal recognition but struggle with the non-literal, symbolic, and common-sense reasoning required to "read" an advertisement. Understanding an ad requires answering two fundamental questions: What should I do? and Why should I do it? This research provides the framework and data to move computer vision toward this higher-level cognitive task of "ad-understanding."
Persuasive Strategies and Methods
The researchers defined a taxonomy of how advertisements function, identifying nine key persuasive strategies. These include:
- Symbolism: Using one object to represent a concept (e.g., a dove for peace).
- Physical Processes: Implying motion or dynamic change to suggest a consequence.
- Atypical Objects: Utilizing non-photorealistic or morphed objects to grab attention.
- Contrast: Highlighting the difference between two states to emphasize a product's benefit.
To evaluate machine performance, several baseline models were implemented:
- Visual Question Answering (VQA): A combined LSTM and VGGNet architecture to predict answers to "Why should I [Action]?" questions.
- Topic/Sentiment Classification: Deep ResNets used to categorize the industry (e.g., Beauty, Electronics) and the intended emotional response.
- Symbolism Detection: An attention-based model designed to locate "signifiers" (physical objects) and link them to "signifieds" (abstract concepts).
Datasets
The study introduces two major benchmarks:
- Image Ad Dataset: Comprising 64,832 images, this dataset includes 202,090 "action-reason" QA pairs and over 64,000 symbolic annotations with bounding boxes.
- Video Ad Dataset: Comprising 3,477 videos, these are annotated for topic, sentiment, "funniness," and "excitement," acknowledging that video ads often rely on temporal engagement and humor.
Key Insights and Results
- The Challenge of Reasoning: While models can often identify the "What" (e.g., "buy this car"), they struggle significantly with the "Why" (e.g., "because it is environmentally friendly"). The baseline model achieved only 11.48% accuracy on open-ended "Why" questions, rising to 48.45% when answers were clustered into 30 prototype reasons.
- Atypicality Breaks Detectors: Ads frequently use morphed objects (like a cow made of kitchen utensils). Because these deviate from the "standard" versions in datasets like ImageNet, traditional object detectors often fail.
- Constructed Sentiment: Ad sentiment is often "constructed" rather than "natural." A disturbing image in a public service announcement might be a tool for a positive cause, making general-purpose sentiment models perform poorly.
- Symbolism as a Bridge: Explicitly detecting symbols (e.g., recognizing that "blood" signifies "injury") slightly improves performance on question-answering tasks, suggesting symbolic decoding is a necessary step for full understanding.
Conclusion and Future Work
Current models remain "black boxes" that lack true human-like reasoning. Future progress in ad-understanding will require the incorporation of external knowledge bases to decode cultural symbols, multimodal integration (such as audio analysis in video), and the development of models capable of explaining the logical leap between a symbolic image and its persuasive intent.
Summary updated