Abstract
Automated moderation of online political advertisements is a critical challenge for digital platforms. Identifying political posters is difficult because neutral news imagery often shares visual characteristics with promotional campaign materials. This paper presents a multimodal deep learning approach that fuses visual appearance features with textual information extracted via Optical Character Recognition (OCR). By combining a ResNeXt-50 architecture with text-derived vectors, the proposed model achieves 96.8% accuracy, significantly outperforming single-modality baselines.
Introduction
Online political advertisements are subject to complex regulatory requirements, ranging from mandatory disclosure of origins to outright bans on certain platforms. However, distinguishing a "political poster" from a neutral news photograph is a non-trivial task for computer vision. While both may feature politicians in suits, flags, or podiums, a poster is defined by its promotional intent—often signaled by specific slogans or calls to action. Traditional image classifiers frequently struggle with this distinction due to the high degree of visual overlap between categories.
Proposed Method: Appearance-Text Fusion
To address the limitations of visual-only models, we propose an Appearance-Text Fusion architecture that processes images through two parallel streams.
- Appearance Stream: A ResNeXt-50 (32x4d) model is utilized to extract a 2048-dimensional feature vector. This stream captures high-level visual context such as political rallies, crowds, and symbolic backgrounds.
- Textual Stream: Text is extracted using the Google Cloud Vision API and converted into a vector via one-hot encoding based on a 50,000-word dictionary.
- Fusion Layer: The appearance and text vectors are concatenated. To balance the influence of the modalities, the text vector is scaled by a factor of $k=0.5$. The resulting multimodal vector is processed by a 3-layer fully connected neural network to generate the final classification.
Dataset and Experimental Setup
The study introduces a novel, human-labeled dataset specifically curated for political poster identification.
The dataset consists of 13,000 images, including 3,000 explicitly labeled political posters. To ensure robustness, the researchers included 10,000 negative samples: 8,200 natural images from ImageNet and 1,800 non-political posters (e.g., entertainment advertisements). This variety prevents the model from simply learning to identify "any poster" or "any politician."
Results and Discussion
Experimental results demonstrate the clear advantage of multimodal fusion. While the appearance-only ResNeXt model achieved 85.1% accuracy and the text-only model reached 83.9%, the RT 3-L Fusion Model achieved a peak accuracy of 96.8%.
Key insights from the evaluation include:
- Complementary Modalities: Appearance features identify the context (e.g., a podium), while text features identify the intent (e.g., "Vote 2020").
- Mitigating False Positives: Visual-only models often fall into the "Suit and Flag" trap, flagging any formal political photo as a poster. The fusion model corrects this by verifying the presence of campaign-specific keywords like "President" or "2020."
- Training Efficiency: The fusion model converged nearly twice as fast as single-modality models, suggesting that the combined feature space provides a more informative gradient for optimization.
Conclusion
The integration of visual and textual data provides a robust solution for the automated identification of political advertisements. While future work will address dictionary scaling and multi-language support, this approach offers a vital tool for platforms navigating the complexities of political ad moderation.
Summary updated