The ALOS Dataset for Advert Localization in Outdoor Scenes

Soumyabrata Dev, Murhaf Hossari, Matthew Nicholson, Killian McCabe, Atul Nautiyal, Clare Conran, Jian Tang, Wei Xu

Abstract

The rapid increase in the number of online videos provides marketing and advertising agents ample opportunities to reach their audience. Product placement, or embedded marketing, involves integrating new advertisements into existing ones within videos. This requires accurate localization of adverts in image frames. We propose and release ALOS (Advert Localization in Outdoor Scenes), the first large-scale dataset of advertisement billboards in outdoor scenes, and benchmark state-of-the-art semantic segmentation algorithms on it.

Introduction

In the domain of "embedded marketing," advertisers often seek to replace existing physical billboards in video content with digital alternatives during post-production. Achieving this seamlessly requires identifying the precise coordinates of the original billboard across every frame. While manual localization is possible, it represents a significant bottleneck for large-scale video processing. Machine learning, specifically deep neural networks, offers a scalable solution but requires vast amounts of high-quality training data.

Example of a typical outdoor scene from the ALOS dataset featuring a billboard.

Prior to this work, no large-scale dataset focused specifically on the precise localization of billboards in diverse outdoor environments existed. General object detection datasets often lack the "four-corner" precision necessary for the perspective warping required in realistic ad replacement.

The ALOS Dataset

The ALOS dataset consists of 8,065 images sourced from Mapillary, a crowdsourcing platform for geotagged street-view imagery. To ensure high quality, the authors manually filtered images for sufficient contrast, legible text, and a minimum resolution of 800x600 pixels.

Every billboard in the dataset was manually annotated as a four-sided convex polygon. This geometric precision is vital for calculating the homography needed to realistically overlay new images onto slanted surfaces in 3D space. The dataset captures various real-world challenges, including partial occlusions and off-screen clipping, as shown in the statistical distributions below.

Statistical distribution of billboard coverage, counts, occlusion states, and off-screen occurrences in the ALOS dataset.

Methodology

The task was framed as a semantic segmentation problem, where models label each pixel as either "billboard" or "background." Three architectures were benchmarked to establish a baseline:

  1. Fully Convolutional Network (FCN): A classic model that outputs spatial maps by replacing fully connected layers with convolutional ones.
  2. Pyramid Scene Parsing Network (PSPNet): A complex model designed to capture global context through multi-scale feature pooling.
  3. U-Net: A symmetric encoder-decoder network originally designed for medical imaging, known for precise localization.

Results and Discussion

Models were evaluated using Pixel Accuracy and Mean Intersection over Union (mIOU). Surprisingly, the lightweight FCN outperformed the more complex PSPNet.

Model Pixel Accuracy Mean IOU Frequency Weighted IOU
FCN 0.962 0.638 0.937
PSPNet 0.554 0.304 0.521
U-Net 0.721 0.432 0.689

The failure of PSPNet to converge effectively suggests that its high complexity may be counterproductive for the specific geometric nature of billboard localization. Furthermore, the results highlight a "definition gap": general-purpose datasets often include small shop signs as billboards, which acts as noise for advertising technology. Specialized datasets like ALOS are therefore essential for high-fidelity visual results.

Conclusion and Future Work

ALOS provides a "gold standard" for training models in outdoor advert localization. However, it is currently limited to vehicle-perspective outdoor scenes. Future work will expand the dataset to include indoor environments (e.g., malls, transit hubs) and dynamic perspectives with motion blur. The authors also suggest that developing bespoke, shallow networks tailored for this task could further improve performance over off-the-shelf models.

Summary updated

Abstract

The rapid increase in the number of online videos provides the marketing and advertising agents ample opportunities to reach out to their audience. One of the most widely used strategies is product placement, or embedded marketing, wherein new advertisements are integrated seamlessly into existing advertisements in videos. Such strategies involve accurately localizing the position of the advert in the image frame, either manually in the video editing phase, or by using machine learning frameworks. However, these machine learning techniques and deep neural networks need a massive amount of data for training. In this paper, we propose and release the first large-scale dataset of advertisement billboards, captured in outdoor scenes. We also benchmark several state-of-the-art semantic segmentation algorithms on our proposed dataset.
BibTeX
@misc{dev2019alos,
  author = {Soumyabrata Dev and Murhaf Hossari and Matthew Nicholson and Killian McCabe and Atul Nautiyal and Clare Conran and Jian Tang and Wei Xu},
  title = {{The ALOS Dataset for Advert Localization in Outdoor Scenes}},
  year = {2019},
  eprint = {1904.07776},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.07776},
}
Figure