Early automated systems for identifying online harms—ranging from fake news and clickbait to cyberbullying and propaganda—relied almost exclusively on natural language processing (NLP). By analyzing word frequencies, syntactic patterns, and semantic embeddings, these models learned to flag deceptive or harmful text.
However, modern online deception is rarely just textual. It is inherently multimodal, combining text with images, video, infographics, and memes. Relying solely on text creates a fundamental blind spot: a model cannot evaluate the truth of a claim if the deception lies entirely within the visual channel, or in the friction between the text and the image.
The Blind Spot of Unimodal Text Analysis
When a system restricts its view to text, it ignores how visual assets are weaponized to manipulate readers. For instance, clickbait often relies on misleading thumbnails or edited images to create a curiosity gap that text alone cannot replicate. As highlighted by Chongyang Shi et al. in Multiview Clickbait Detection via Jointly Modeling Subjective and Objective Preference, text-only frameworks fail to capture how visual and textual cues work together to deceive users.
This limitation extends across the entire spectrum of online misinformation:
- Disinformation Campaigns: In public health crises, such as the COVID-19 pandemic, propaganda frequently spreads via infographics and videos rather than plain text. Yida Mu et al. demonstrated in VaxxHesitancy: A Dataset for Studying Hesitancy towards COVID-19 Vaccination on Twitter that text-based analysis overlooks the critical role these visual media play in shaping public attitudes and spreading vaccine hesitancy.
- Fact-Checking and Verification: Automated fact-checking pipelines struggle to prioritize what to verify when they cannot see the full context. Maram Hasanain et al. noted in Overview of the CLEF-2024 CheckThat! Lab Task 1 on Check-Worthiness Estimation of Multigenre Content that text-only models remain blind to a significant portion of modern deceptive content, such as manipulated videos and memes.
- Online Abuse and Cyberbullying: Traditional cyberbullying detection models rely heavily on psychometric and linguistic features extracted from text (Parma Nand et al. in “How Bullying is this Message?”: A Psychometric Thermometer for Bullying; Qianjia Huang et al. in Cyberbullying Intervention Based on Convolutional Neural Networks). Yet, harmful expressions are increasingly embedded within multimodal formats. As shown by Pablo Cordon et al. in I2C at SemEval-2022 Task 5: Identification of misogyny in internet memes, text-only approaches leave a massive gap when addressing abuse disguised within internet memes, where the text might seem benign out of context but becomes toxic when paired with a specific image.
Even highly sophisticated language models designed to unify multiple text-based tasks underperform in the wild because they lack grounding in other modalities. For example, while the unified model developed by Nayeon Lee et al. in On Unifying Misinformation Detection generalizes well across text domains, the authors explicitly noted that neglecting visual and social propagation patterns limits its comprehensive detection capabilities.
The Core Mechanism: Cross-Modal Inconsistency
Why does adding images or video make a difference? The power of multimodal detection does not simply come from having more data; it comes from modeling the relationship between modalities.
In legitimate news, text and images typically reinforce one another. In fake news, they are often mismatched—a real image might be paired with a completely fabricated headline, or a manipulated image might be used to support a factual text block. Text-only models are fundamentally incapable of detecting these cross-modal inconsistencies.
[ Text Input ] ---------> NLP Encoder ---------\
+---> [ Fusion Layer ] ---> Is it Misinformation?
[ Image Input ] --------> Vision Encoder ------/ |
v
Detects Cross-Modal Friction
(e.g., Text and Image don't match)
To exploit this signal, researchers have shifted from simply concatenating text and image features to explicitly measuring cross-modal friction. Lianwei Wu et al. demonstrated in Unified Evidence Enhancement Inference Framework for Fake News Detection that text-only models cannot account for visual deceptions or deepfakes. By contrast, multimodal frameworks can cross-reference specific text fragments against visual evidence to expose structural contradictions.
Rather than just looking for similarities between text and graphics, advanced frameworks actively hunt for discrepancies. In Multimodal Inverse Attention Network with Intrinsic Discriminant Feature Exploitation for Fake News Detection, Tianlin Zhang et al. designed an inverse attention mechanism specifically to isolate features where the text and image disagree. Similarly, Linmei Hu et al. introduced an image-text matching-aware co-attention network in Multimodal matching-aware co-attention networks with mutual knowledge distillation for fake news detection to force the model to learn fine-grained alignments between textual phrases and visual regions, flagging content when the two modalities drift apart.
Structural and Architectural Bottlenecks
Transitioning from text-only to multimodal analysis introduces severe data and architectural challenges that the research community is actively trying to solve.
1. Dataset Limitations
Historically, fake news benchmarks were entirely textual. To build robust multimodal models, the field required new data paradigms. Kai Nakamura et al. addressed this in Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection by releasing a massive, multimodal dataset from Reddit, allowing models to learn the complex interplay between text and imagery across fine-grained categories of misinformation.
2. Platform-Level Text Scarcity
On modern social platforms like TikTok or Instagram, text is often minimal or entirely absent, rendering traditional NLP pipelines obsolete. To detect deceptive behavior on these platforms, models must bypass text altogether. For instance, Jiacheng Li et al. showed in Who Are Controlled by The Same User? Multiple Identities Deception Detection via Social Interaction Activity (Student Abstract) that tracking the structural entropy of social interactions allows for the detection of malicious "sockpuppet" accounts across video-dominated ecosystems without relying on any linguistic features.
3. The Single-Image Bottleneck
Even current multimodal frameworks suffer from an oversimplification: they are typically designed to process exactly one image per article or post. As noted independently by both Tianlin Zhang et al. and Linmei Hu et al., real-world social media posts frequently contain multi-image galleries or sequential carousels. Randomly selecting a single image drops vital context, meaning that extending architectures to handle multi-image sequences remains a critical frontier in fully overcoming the constraints of unimodal analysis.
Key Foundational Work
Overview and Comparison of Plagiarism Detection Tools (2011) — contributed 1 concept elements to this direction.
Linguistic and Statistical Traits Characterising Plagiarism (2012) — contributed 1 concept elements to this direction.