Introduction
Content moderation on social media platforms presents a significant scalability challenge, particularly for languages other than English. The GermEval 2021 shared task addresses this by focusing on the classification of German Facebook comments. Researchers from IIIT Hyderabad (Precog-LTRC-IIITH) developed a robust system to tackle three specific subtasks: identifying offensive language (Subtask A: Toxic), identifying comments that promote fruitful information exchange (Subtask B: Engaging), and identifying comments that assert facts (Subtask C: Fact-Claiming).
Methodology
The authors adopted a hybrid approach that integrates modern deep learning architectures with traditional natural language processing (NLP) techniques. This strategy was designed to handle the inherent messiness and data sparsity of social media text.
Feature Engineering
A core component of the system was the engineering of 20 specific features. These were categorized into stylistic features (e.g., length, punctuation, emoji distribution) and linguistic features (e.g., sentiment scores via SentiWS, readability indices, and grammatical markers like verb tenses). By using scikit-learn to determine feature importance, the researchers demonstrated that these hand-crafted markers provide critical signals that raw text models might miss.
Model Architectures and Augmentation
The team experimented with several pre-trained Transformer models, including BERT, Electra, ConvBERT, and XLM-Roberta. To enhance these models, they attached specialized "heads":
- CNN Heads: Captured local patterns from the last four hidden layers.
- Capsule Networks: Preserved spatial and proportional relationships between words, which proved effective for text classification.
- RNN-GRU Heads: Used alongside FastText and GloVe embeddings to manage out-of-vocabulary (OOV) words common in social media.
To address data sparsity, the researchers employed back-translation. Training data was translated from German to English and back to German, creating paraphrased versions that expanded the training set and improved model generalization.
Results
The final system utilized a two-level ensemble, averaging predictions across different architectures and training folds. This ensemble approach consistently outperformed the standard fine-tuned BERT baseline across all subtasks.
The ensemble achieved its highest performance in Subtask C (Fact-Claiming) with an F1 score of 73.91. Individual models also showed strength; for instance, the Electra model with a Capsule Net head was one of the strongest individual performers, particularly in identifying toxicity.
Discussion and Future Work
The study yielded several key insights:
- Feature Engineering Value: Hand-crafted features remain highly relevant in the era of deep learning, especially when training data is limited.
- Truncation Challenges: Standard Transformers often truncate long comments at 512 tokens. The authors noted that many misclassifications occurred in longer texts, suggesting that future work should utilize models like Longformers to capture the full context.
- Contextual Limitations: Models evaluating comments in isolation often struggle with sarcasm or thread-level context. Incorporating parent posts or preceding replies is a promising direction for reducing false positives.
By combining diverse model architectures with linguistic insights, the Precog-LTRC-IIITH team demonstrated a highly effective framework for multilingual social media moderation.
Summary updated