The Art of Embedding Fusion: Optimizing Hate Speech Detection

Mohammad Aflah Khan, Neemesh Yadav, Mohit Jain, Sanyam GoyalSubmission to Tiny Papers @ ICLR

Does combining multiple AI "perspectives" make for better hate speech detection? This paper explores "embedding fusion"—the art of merging the mathematical representations from different pre-trained models like BERT, HateBERT, and BERTweet. The authors find that while combining these models offers a slight performance boost, the specific method of fusion matters less than the fact that you're using domain-specific models in the first place.

What Problem Does This Paper Address?

Hate speech is notoriously difficult for AI to detect because it relies heavily on linguistic nuances, sarcasm, and evolving social contexts. While general-purpose models like BERT are powerful, researchers have developed domain-specific versions—like HateBERT (trained on abusive Reddit comments) and BERTweet (trained on English tweets)—to better capture these subtleties.

The challenge is that we don't know the best way to leverage these models together. Should we just pick the best one? Or is there a "synergy" to be found by combining their internal representations (embeddings)? If we do combine them, how should we do it? This paper systematically tests different "fusion" strategies to see if the extra computational cost of running multiple models actually pays off in accuracy.

Methods

The researchers extracted "pooler outputs" (dense vector representations of sentences) from three models: BERT, HateBERT, and BERTweet. They then tested five distinct strategies to fuse these vectors before feeding them into a Multi-Layer Perceptron (MLP) classifier:

  1. Addition: Simply adding the vectors together element-wise.
  2. Multiplication: Element-wise multiplication.
  3. Concatenation: Stacking the vectors end-to-end (e.g., a 768-dim vector becomes 1536 or 2304).
  4. Interleaving: "Zipping" the vectors together (e.g., taking the first element of each, then the second, etc.).
  5. Random Interleaving: A baseline where dimensions are shuffled randomly, destroying the alignment between models.

They used five-fold cross-validation and grid search to ensure the results were robust and not just a fluke of specific hyperparameter settings.

Datasets

The study evaluated these methods across three diverse benchmarks:

  • OLID: A standard dataset for offensive vs. non-offensive Twitter posts.
  • Latent Hatred: Focuses on the tricky distinction between implicit hate, explicit hate, and non-hate.
  • DynaHate: A more recent, challenging dataset created using a "human-in-the-loop" process to generate sentences specifically designed to fool AI.

Key Insights and Ideas

Accuracy across different fusion models on the DynaHate dataset
  • Diminishing Returns on Complexity: The most surprising finding is that the specific fusion method (concatenation vs. interleaving vs. multiplication) has a marginal effect on the final results. As long as you aren't using "Random Interleaving," most methods perform similarly.
  • The "Addition" Hack for Efficiency: Concatenation and Interleaving increase the input size of the final classifier by 2x or 3x, which increases memory and compute requirements. However, Addition—which keeps the vector size the same—performed nearly as well. This makes it a "best-buy" for low-compute environments.
  • Domain Knowledge is King: Models trained on specific data (HateBERT and BERTweet) consistently outperformed the general BERT model. Combining them is better than using one, but the performance gain is "slight" compared to the jump from a general model to a domain-specific one.
  • Why Randomness Fails: Random interleaving performed significantly worse because it makes the embeddings "degenerate." In a structured embedding, specific dimensions often correspond to specific semantic features; shuffling them across different models creates mathematical noise that the classifier struggles to decode.

Key Results

The performance across all combinations was very tight, but a few patterns emerged:

Top 5 model combinations by accuracy on the DynaHate dataset
  • Top Performer: On the DynaHate dataset, the triple combination of BERT + BERTweet + HateBERT (Interleaved) achieved the highest accuracy of 0.716.
  • Consistency: On the Latent Hatred dataset, the top five combinations all hovered around 0.700 to 0.703 accuracy.
  • The Baseline: Random interleaving consistently dropped performance by roughly 10-15% (e.g., dropping from ~0.71 to ~0.55 on DynaHate), proving that the structural alignment of embeddings is crucial.

Limitations and Future Work

As an ICLR "Tiny Paper," the scope was intentionally focused. The authors acknowledge that while combining embeddings helps, it comes at a high computational cost (running three large models instead of one).

Future work could explore:

  • Gated Fusion: Using an attention mechanism to let the model "decide" which embedding to trust more for a specific sentence.
  • Dimensionality Reduction: Testing if techniques like PCA could compress fused embeddings without losing the slight accuracy gains.
  • Broader Model Sets: Testing newer, even larger models or different architectures (like RoBERTa or DeBERTa) to see if the "fusion" rules still hold.

Summary updated

Abstract

Hate speech detection is a challenging natural language processing task that requires capturing linguistic and contextual nuances. Pre-trained language models (PLMs) offer rich semantic representations of text that can improve this task. However there is still limited knowledge about ways to effectively combine representations across PLMs and leverage their complementary strengths. In this work, we shed light on various combination techniques for several PLMs and comprehensively analyze their effectiveness. Our findings show that combining embeddings leads to slight improvements but at a high computational cost and the choice of combination has marginal effect on the final outcome.
BibTeX
@misc{khan2023art,
  author = {Mohammad Aflah Khan and Neemesh Yadav and Mohit Jain and Sanyam Goyal},
  title = {{The Art of Embedding Fusion: Optimizing Hate Speech Detection}},
  year = {2023},
  eprint = {2306.14939},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.14939},
}
Figure