Political Posters Identification with Appearance-Text Fusion

Xuan Qin, Meizhu Liu, Yifan Hu, Christina Moo, Christian M. Riblet, Changwei Hu, Kevin Yen, Haibin LingarXiv 2020 · arXiv

Abstract

Automated moderation of online political advertisements is a critical challenge for digital platforms. Identifying political posters is difficult because neutral news imagery often shares visual characteristics with promotional campaign materials. This paper presents a multimodal deep learning approach that fuses visual appearance features with textual information extracted via Optical Character Recognition (OCR). By combining a ResNeXt-50 architecture with text-derived vectors, the proposed model achieves 96.8% accuracy, significantly outperforming single-modality baselines.

Introduction

Online political advertisements are subject to complex regulatory requirements, ranging from mandatory disclosure of origins to outright bans on certain platforms. However, distinguishing a "political poster" from a neutral news photograph is a non-trivial task for computer vision. While both may feature politicians in suits, flags, or podiums, a poster is defined by its promotional intent—often signaled by specific slogans or calls to action. Traditional image classifiers frequently struggle with this distinction due to the high degree of visual overlap between categories.

Proposed Method: Appearance-Text Fusion

To address the limitations of visual-only models, we propose an Appearance-Text Fusion architecture that processes images through two parallel streams.

Architecture of the Appearance-Text Fusion model, combining visual features and OCR-extracted text.
  1. Appearance Stream: A ResNeXt-50 (32x4d) model is utilized to extract a 2048-dimensional feature vector. This stream captures high-level visual context such as political rallies, crowds, and symbolic backgrounds.
  2. Textual Stream: Text is extracted using the Google Cloud Vision API and converted into a vector via one-hot encoding based on a 50,000-word dictionary.
  3. Fusion Layer: The appearance and text vectors are concatenated. To balance the influence of the modalities, the text vector is scaled by a factor of $k=0.5$. The resulting multimodal vector is processed by a 3-layer fully connected neural network to generate the final classification.

Dataset and Experimental Setup

The study introduces a novel, human-labeled dataset specifically curated for political poster identification.

Examples of political posters and neutral political imagery illustrating the challenge of visual similarity.

The dataset consists of 13,000 images, including 3,000 explicitly labeled political posters. To ensure robustness, the researchers included 10,000 negative samples: 8,200 natural images from ImageNet and 1,800 non-political posters (e.g., entertainment advertisements). This variety prevents the model from simply learning to identify "any poster" or "any politician."

Results and Discussion

Experimental results demonstrate the clear advantage of multimodal fusion. While the appearance-only ResNeXt model achieved 85.1% accuracy and the text-only model reached 83.9%, the RT 3-L Fusion Model achieved a peak accuracy of 96.8%.

Key insights from the evaluation include:

  • Complementary Modalities: Appearance features identify the context (e.g., a podium), while text features identify the intent (e.g., "Vote 2020").
  • Mitigating False Positives: Visual-only models often fall into the "Suit and Flag" trap, flagging any formal political photo as a poster. The fusion model corrects this by verifying the presence of campaign-specific keywords like "President" or "2020."
  • Training Efficiency: The fusion model converged nearly twice as fast as single-modality models, suggesting that the combined feature space provides a more informative gradient for optimization.

Conclusion

The integration of visual and textual data provides a robust solution for the automated identification of political advertisements. While future work will address dictionary scaling and multi-language support, this approach offers a vital tool for platforms navigating the complexities of political ad moderation.

Summary updated

Abstract

In this paper, we propose a method that efficiently utilizes appearance features and text vectors to accurately classify political posters from other similar political images. The majority of this work focuses on political posters that are designed to serve as a promotion of a certain political event, and the automated identification of which can lead to the generation of detailed statistics and meets the judgment needs in a variety of areas. Starting with a comprehensive keyword list for politicians and political events, we curate for the first time an effective and practical political poster dataset containing 13K human-labeled political images, including 3K political posters that explicitly support a movement or a campaign. Second, we make a thorough case study for this dataset and analyze common patterns and outliers of political posters. Finally, we propose a model that combines the power of both appearance and text information to classify political posters with significantly high accuracy.
BibTeX
@misc{qin2020political,
  author = {Xuan Qin and Meizhu Liu and Yifan Hu and Christina Moo and Christian M. Riblet and Changwei Hu and Kevin Yen and Haibin Ling},
  title = {{Political Posters Identification with Appearance-Text Fusion}},
  year = {2020},
  eprint = {2012.10728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.10728},
}
Figure