Abstract
This study investigates the performance disparity in multimodal hate speech detection between video content and image-text memes. While video-based detection has seen significant progress through simple embedding fusion, memes remain a formidable challenge due to their non-additive semantic nature. We evaluate various fusion paradigms, including a novel sequential attention approach (MO-Hate), across the HateMM and Hateful Memes Challenge (HMC) datasets. Our findings reveal that while improved Automatic Speech Recognition (ASR) significantly boosts video detection performance, current models struggle to decode the interdependent relationships in memes, particularly when faced with benign confounders.
Introduction
Hate speech has evolved beyond simple text, propagating through complex multimodal formats such as videos with synchronized audio-visual cues and memes where meaning emerges from the interplay between an image and its caption. The core challenge lies in multimodal fusion: effectively combining signals from disparate encoders (text, vision, audio) into a single coherent understanding. This paper explores why multimodal hate detection is nearing a "solved" state for video content while remaining a significant hurdle for memes.
Methodology
The research evaluates two primary fusion paradigms across state-of-the-art encoders:
- Simple Embedding Fusion (Early Fusion): Pre-trained embeddings from models like BERT (text), CLIP (images), or Wav2Vec2 (audio) are concatenated or combined via element-wise products before classification.
- Modality Order-Aware Fusion (MO-Hate): A sophisticated "late fusion" approach using a modified BART architecture. It processes modalities sequentially (e.g., Text → Audio → Visual), employing cross-modal attention to allow each modality to influence the subsequent representation.
A significant technical improvement was the integration of OpenAI’s Whisper (tiny) for the ASR pipeline in video datasets, replacing older tools like Vosk to provide higher-quality text transcripts.
Datasets and the Challenge of Confounders
The study utilizes two distinct benchmarks to highlight the differences in multimodal reasoning:
- HateMM (Video-based): 1,083 labeled videos from BitChute, featuring visual frames, audio tracks, and transcripts.
- Hateful Memes Challenge (HMC): ~10,000 memes designed by Facebook AI to be intentionally difficult. These include "benign confounders"—harmless images paired with specific text to create a hateful message, or vice versa.
Analysis of Multimodal Fusion
The Synchronicity Advantage in Video
In video content, modalities are often redundant or reinforcing. A slur shouted in an audio track is typically mirrored in the transcript. Simple fusion excels here because it only needs to detect the strongest signal rather than reason about complex interactions.
The "Meme Wall"
Memes are semantically "non-additive." The hateful intent is rarely found in the image or text alone; it exists in the spark between them. Current models often rely too heavily on "unimodal priors"—guessing based on just one modality. When faced with benign confounders, where the image is harmless but the combination is toxic, model accuracy plummets.
Key Results
- State-of-the-Art for Video: On the HateMM dataset, the best-performing model (HCC1) achieved a macro-F1 score of 0.848, representing a 9.9 percentage point improvement over previous benchmarks. This gain was largely driven by the switch to Whisper ASR.
- The Meme Struggle: On the HMC dataset, the best model (MBD1) reached an AUROC of 0.628. While this outperforms zero-shot Vision-Language Models, it lags significantly behind specialized HMC leaderboard models (~0.89 AUROC), confirming that sequential attention is insufficient for meme decoding.
Conclusion and Future Work
While video hate detection benefits from modality redundancy and high-quality ASR, memes require deeper semantic reasoning. Future research should move toward unified frameworks incorporating object detection and captioning to explicitly identify visual components (e.g., "a picture of a cop") and reason about their interaction with text. Additionally, improving video detection will involve identifying specific timestamps of hateful content rather than aggregate video labeling.
Summary updated