Abstract
The rapid increase in the number of online videos provides marketing and advertising agents ample opportunities to reach their audience. Product placement, or embedded marketing, involves integrating new advertisements into existing ones within videos. This requires accurate localization of adverts in image frames. We propose and release ALOS (Advert Localization in Outdoor Scenes), the first large-scale dataset of advertisement billboards in outdoor scenes, and benchmark state-of-the-art semantic segmentation algorithms on it.
Introduction
In the domain of "embedded marketing," advertisers often seek to replace existing physical billboards in video content with digital alternatives during post-production. Achieving this seamlessly requires identifying the precise coordinates of the original billboard across every frame. While manual localization is possible, it represents a significant bottleneck for large-scale video processing. Machine learning, specifically deep neural networks, offers a scalable solution but requires vast amounts of high-quality training data.
Prior to this work, no large-scale dataset focused specifically on the precise localization of billboards in diverse outdoor environments existed. General object detection datasets often lack the "four-corner" precision necessary for the perspective warping required in realistic ad replacement.
The ALOS Dataset
The ALOS dataset consists of 8,065 images sourced from Mapillary, a crowdsourcing platform for geotagged street-view imagery. To ensure high quality, the authors manually filtered images for sufficient contrast, legible text, and a minimum resolution of 800x600 pixels.
Every billboard in the dataset was manually annotated as a four-sided convex polygon. This geometric precision is vital for calculating the homography needed to realistically overlay new images onto slanted surfaces in 3D space. The dataset captures various real-world challenges, including partial occlusions and off-screen clipping, as shown in the statistical distributions below.
Methodology
The task was framed as a semantic segmentation problem, where models label each pixel as either "billboard" or "background." Three architectures were benchmarked to establish a baseline:
- Fully Convolutional Network (FCN): A classic model that outputs spatial maps by replacing fully connected layers with convolutional ones.
- Pyramid Scene Parsing Network (PSPNet): A complex model designed to capture global context through multi-scale feature pooling.
- U-Net: A symmetric encoder-decoder network originally designed for medical imaging, known for precise localization.
Results and Discussion
Models were evaluated using Pixel Accuracy and Mean Intersection over Union (mIOU). Surprisingly, the lightweight FCN outperformed the more complex PSPNet.
| Model | Pixel Accuracy | Mean IOU | Frequency Weighted IOU |
|---|---|---|---|
| FCN | 0.962 | 0.638 | 0.937 |
| PSPNet | 0.554 | 0.304 | 0.521 |
| U-Net | 0.721 | 0.432 | 0.689 |
The failure of PSPNet to converge effectively suggests that its high complexity may be counterproductive for the specific geometric nature of billboard localization. Furthermore, the results highlight a "definition gap": general-purpose datasets often include small shop signs as billboards, which acts as noise for advertising technology. Specialized datasets like ALOS are therefore essential for high-fidelity visual results.
Conclusion and Future Work
ALOS provides a "gold standard" for training models in outdoor advert localization. However, it is currently limited to vehicle-perspective outdoor scenes. Future work will expand the dataset to include indoor environments (e.g., malls, transit hubs) and dynamic perspectives with motion blur. The authors also suggest that developing bespoke, shallow networks tailored for this task could further improve performance over off-the-shelf models.
Summary updated