Precog-LTRC-IIITH at GermEval 2021: Ensembling Pre-Trained Language Models with Feature Engineering

T. H. Arjun, Arvindh A., Ponnurangam KumaraguruGermEval@KONVENS 2021

Introduction

Content moderation on social media platforms presents a significant scalability challenge, particularly for languages other than English. The GermEval 2021 shared task addresses this by focusing on the classification of German Facebook comments. Researchers from IIIT Hyderabad (Precog-LTRC-IIITH) developed a robust system to tackle three specific subtasks: identifying offensive language (Subtask A: Toxic), identifying comments that promote fruitful information exchange (Subtask B: Engaging), and identifying comments that assert facts (Subtask C: Fact-Claiming).

Methodology

The authors adopted a hybrid approach that integrates modern deep learning architectures with traditional natural language processing (NLP) techniques. This strategy was designed to handle the inherent messiness and data sparsity of social media text.

Feature Engineering

A core component of the system was the engineering of 20 specific features. These were categorized into stylistic features (e.g., length, punctuation, emoji distribution) and linguistic features (e.g., sentiment scores via SentiWS, readability indices, and grammatical markers like verb tenses). By using scikit-learn to determine feature importance, the researchers demonstrated that these hand-crafted markers provide critical signals that raw text models might miss.

Feature importance across the three subtasks, highlighting the relevance of stylistic and linguistic markers.

Model Architectures and Augmentation

The team experimented with several pre-trained Transformer models, including BERT, Electra, ConvBERT, and XLM-Roberta. To enhance these models, they attached specialized "heads":

  • CNN Heads: Captured local patterns from the last four hidden layers.
  • Capsule Networks: Preserved spatial and proportional relationships between words, which proved effective for text classification.
  • RNN-GRU Heads: Used alongside FastText and GloVe embeddings to manage out-of-vocabulary (OOV) words common in social media.

To address data sparsity, the researchers employed back-translation. Training data was translated from German to English and back to German, creating paraphrased versions that expanded the training set and improved model generalization.

Results

The final system utilized a two-level ensemble, averaging predictions across different architectures and training folds. This ensemble approach consistently outperformed the standard fine-tuned BERT baseline across all subtasks.

Detailed performance metrics (F1, Precision, Recall) for individual models and the final ensemble submission across Subtasks A, B, and C.

The ensemble achieved its highest performance in Subtask C (Fact-Claiming) with an F1 score of 73.91. Individual models also showed strength; for instance, the Electra model with a Capsule Net head was one of the strongest individual performers, particularly in identifying toxicity.

Discussion and Future Work

The study yielded several key insights:

  1. Feature Engineering Value: Hand-crafted features remain highly relevant in the era of deep learning, especially when training data is limited.
  2. Truncation Challenges: Standard Transformers often truncate long comments at 512 tokens. The authors noted that many misclassifications occurred in longer texts, suggesting that future work should utilize models like Longformers to capture the full context.
  3. Contextual Limitations: Models evaluating comments in isolation often struggle with sarcasm or thread-level context. Incorporating parent posts or preceding replies is a promising direction for reducing false positives.

By combining diverse model architectures with linguistic insights, the Precog-LTRC-IIITH team demonstrated a highly effective framework for multilingual social media moderation.

Summary updated

BibTeX
@inproceedings{arjun2021precog,
  author = {T. H. Arjun and Arvindh A. and Ponnurangam Kumaraguru},
  title = {{Precog-LTRC-IIITH at GermEval 2021: Ensembling Pre-Trained Language Models with Feature Engineering}},
  booktitle = {GermEval@KONVENS},
  year = {2021},
}
Figure