Training Language Models under Resource Constraints for Adversarial Advertisement Detection

Eshwar Shamanna Girishekar, Shiv Surya, Nishant Nikhil, Dyut Kumar Sil, Sumit Negi, Aruna RajanNAACL 2021 · DOI

Introduction

Detecting prohibited or "adversarial" advertisements on e-commerce platforms is a critical "needle in a haystack" problem. Online marketplaces must filter out ads containing prohibited content—such as adult material, illegal products, or hate speech—to maintain customer trust and comply with regulatory standards. However, this task faces significant hurdles: extreme class imbalance (prohibited ads are rare), resource constraints in non-English languages, and the adversarial nature of advertisers who use creative language to bypass filters.

This research demonstrates how to significantly boost detection performance by combining weak supervision, curriculum learning, and strategic multilingual training for BERT-based models, particularly when labeled data is scarce and human annotation is difficult to scale.

Methodology

The researchers explored three primary strategies to overcome data scarcity:

Semi-Supervised and Semi-Weak Supervision

A "teacher-student" approach was employed where a BERT teacher model, trained on limited human-labeled ("strong") data, predicts labels for a large pool of unlabeled data. The highest-scoring samples are used to pre-train a "student" model before final fine-tuning on strong data. The Semi-Weak Supervision (SWS) variant incorporates keyword-based filtering to source the initial weak data.

Curriculum Learning

Data was introduced to the model progressively rather than all at once:

  • Hardness-based: A simple word-embedding model (BOE_LIN) scores sample difficulty. The model learns from "easy" samples first.
  • Lexical Similarity-based: For multilingual models (M-BERT), data is fed in order of lexical similarity to the target language, using scores derived from Wikipedia.

Data Augmentation

High-resource English (EN) labeled data was translated into target languages like German (DE) and French (FR) to supplement training sets.

The study utilized internal Amazon advertising data across five semantic categories of prohibited content (CAT1–CAT5) and five languages: English (EN), German (DE), French (FR), Spanish (ES), and Italian (IT).

Key Results and Insights

The evaluation focused on Precision at high Recall thresholds and the True Negative Rate (TNR)—the percentage of "clean" ads correctly approved.

Precision Gains through Weak Supervision

One of the most significant findings is the impact of "noisy" data. Weak supervision yielded an average precision gain of 10.88% across languages. In English, the SWS approach improved precision by over 40% for specific categories like CAT1 compared to the baseline.

Precision improvement of B_SS and B_SWS methods over baseline across five categories.

Success in Low-Resource Scenarios

For German and French, curriculum strategies using translated data improved the average TNR by 24.25%. Training on available data from all five languages provided an average gain of 24.32% in TNR over the baseline. Ablation studies confirmed that an "easy-to-hard" curriculum is statistically superior to random or "hard-to-easy" ordering.

Performance comparison (TNR and Precision) across different model architectures and ablation studies for German (DE) and French (FR).

The Importance of Learning Order

Simply adding translated data is insufficient. Using a curriculum helps the model converge better and avoids "semantic drift" caused by imperfect translations. Furthermore, M-BERT performs better when "warmed up" on languages with high lexical proximity to the target language.

Conclusion and Future Work

While these methods show great promise, challenges remain. "Semantic drift" from imperfect translations can confuse the model's understanding of subtle prohibited content. Additionally, sourcing initial weak data still depends on manually curated keyword lists. Future research will explore more advanced data augmentation and refine the transition between synthetic translated data and real-world labeled data.

Summary updated

BibTeX
@inproceedings{girishekar2021training,
  author = {Eshwar Shamanna Girishekar and Shiv Surya and Nishant Nikhil and Dyut Kumar Sil and Sumit Negi and Aruna Rajan},
  title = {{Training Language Models under Resource Constraints for Adversarial Advertisement Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-industry.35},
}
Figure