TL;DR
This paper tackles the challenge of "adjacent marketing" in image advertising by mining relationships between products from a large collection of video ads. They introduce a novel Multi-Modal Dirichlet Process Mixture Sets (MoM-DPM Sets) model that unifies topic discovery and ad ranking, enabling the suggestion of complementary products (e.g., car insurance for car ads) beyond direct visual matches.
What Problem Does This Paper Address?
The field of image advertising is still nascent, with most existing approaches focusing on directly matching an ad to a query image. This often fails to capture the broader context of consumer needs and the relationships between different products. "Adjacent marketing" aims to address this by suggesting complementary items that a user might be interested in, much like a car dealership might suggest insurance or accessories after a car purchase. The core challenge is to automatically discover these "adjacent markets" from large-scale data, especially when the visual and textual similarities between the query and the adjacent product are not immediately obvious.
Methods
The authors propose a novel Multi-Modal Dirichlet Process Mixture Sets (MoM-DPM Sets) model. This model is designed to address four key challenges:
- Discovering latent topics: It identifies underlying themes or concepts shared across different video frames.
- Automatic topic number determination: It doesn't require pre-defining the number of topics.
- Leveraging multi-modal features: It effectively combines both visual and textual information from video frames.
- Unifying topic mining and ad ranking: It integrates the process of discovering these themes with the task of ranking relevant ads.
The system overview involves an offline stage where video keyframes are extracted and auto-tagged. The online stage then performs:
- Multimodal search: Given a query image, it searches for similar video frames using both visual features and textual information (from auto-tagging, OCR, and surrounding text).
- Search result expansion: To incorporate adjacent market information, retrieved frames are expanded with other frames from the same video. This helps uncover relationships that might not be present in the initially retrieved frames.
- Ad clustering and ranking: The MoM-DPM Sets model is used to learn key topics from the expanded search results and then rank ads based on these learned topics.
Datasets
- Video Ads Collection: Approximately 32,000 videos from YouTube, initiated by 30 popular advertising concepts.
- Key Frames: 327,889 key frames extracted from these videos.
- Ads Database: A separate database of 450 ads used for ranking purposes.
- Query Images: 100 Flickr images used as queries.
The paper does not explicitly state the creation of a new benchmark dataset but rather uses a large-scale collection of video ads for their experiments.
Key Insights and Ideas
- Video ads as a rich source for adjacent marketing: The paper highlights that video ads, with their narrative "story frames" and explicit "ads frames," implicitly encode human knowledge about product relationships and adjacent markets.
- Beyond direct visual similarity: The core innovation is moving beyond simple image-to-image matching to discover complementary products that might have low visual or textual overlap with the query.
- The power of "set-based reasoning": The MoM-DPM Sets model's ability to discover analogical objects (e.g., recognizing "soda" from both Pepsi and Coca-Cola frames) is crucial for identifying broader market categories.
- Unified model for discovery and ranking: Combining topic mining and ad ranking within a single model formulation simplifies the process and potentially improves accuracy by ensuring the discovered topics are directly relevant for ranking.
Key Results
- Consistent outperformance: The proposed approach consistently outperforms baseline methods (Argo and direct match) in terms of average precision at top N results.
- Significant gap for adjacent markets: The gap between their method and baselines, especially the direct match, indicates their success in identifying ads from potential adjacent markets.
- Promising for publishers: While the gap narrows at higher N values (suggesting potential limitations with smaller ad databases), the strong performance in the top 3-5 results is particularly promising for publishers who typically display a limited number of targeted ads.
Limitations and Future Work
- Ad database size: The authors acknowledge that the limited size of their ads database might explain the narrowing gap at higher N values. They suggest that a larger ads database could further improve performance.
- Publisher ad display limits: The paper notes that publishers generally show fewer than five targeted ads, making their strong performance in the top few results particularly relevant and suggesting a promising direction for future research.
- Future work: The paper implicitly suggests exploring larger datasets and potentially more sophisticated multimodal fusion techniques to further enhance the discovery and ranking of adjacent market ads.
Summary updated