Multimodal models are designed under the assumption that fusing text, images, audio, and metadata yields a more holistic understanding of digital content. Intuitively, more data streams should produce better predictions. Yet, a recurring pattern across a wide array of tasks—ranging from meme sentiment analysis to cross-platform fake news detection—reveals a striking counter-intuition: textual signals frequently dominate the learning process, often matching or outperforming complex multimodal architectures.
This phenomenon, known as textual information dominance, suggests that language acts as a highly concentrated, low-noise repository of intent and semantics. In contrast, other modalities often introduce confounding variance that can lead models astray. Understanding why text dominates requires looking at how information is structured, recycled, and deployed across different media.
The Anatomy of Visual Noise and Template Reuse
Why does text regularly outperform rich visual data in tasks like meme analysis? The answer lies in how memes are constructed and consumed.
As Vishal Keswani et al. demonstrate in their work on internet meme sentiment, visual templates are frequently recycled across completely opposing emotional categories. A single image macro can express joy, deep frustration, or irony depending entirely on the overlaid text. Because the pixel distribution has a low correlation with the underlying sentiment, forcing a model to process the image can degrade performance. In their experiments, a simple text-only network relying on word embeddings outperformed state-of-the-art multimodal transformers.
[Image Template: Distracted Boyfriend]
├── Text A: "Me looking at a new framework" ──> Positive Sentiment
└── Text B: "Me ignoring my responsibilities" ──> Negative Sentiment
Visual templates are invariant across conflicting categories, meaning the raw pixels carry zero correlation with the target label. The discriminative signal resides entirely in the text.
This principle extends directly to content moderation. Yehia Elkhatib et al. discovered through randomized controlled trials that a meme's offensive potential is driven almost entirely by its textual message rather than its visual format. When detecting specific harms like misogyny or trolling in memes, text-only models consistently achieve highly competitive or superior baselines compared to multimodal systems (Gagan Sharma et al.; Md Hasan et al.).
As Mayukh Sharma et al. observe, multimodal fusion mechanisms often yield only marginal improvements over strong text-only baselines because the core harmful message is fundamentally linguistic. Harshvardhan Srivastava further notes that the influence of text can vary by sub-type, but it remains the primary anchor for identifying explicit social harms.
Even when training Large Multimodal Models (LMMs), visual data can introduce unwanted variance. Zixi Jia et al. showed that retrieving multimodal examples for in-context learning using text-only queries is more effective than using joint image-text embeddings. Images introduce high-dimensional noise into the embedding space, whereas text queries provide a clean, precise indicator of the underlying logical structure required for a reasoning task.
Semantic Explicitness and Intent Detection
Beyond avoiding noise, text dominates because it explicitly encodes propositional content and intent. In social media monitoring and fact-checking, an image might depict a benign object or a generic scene, but the accompanying text contains the actionable signal.
For instance, when identifying illicit drug dealers on Instagram, Chuanbo Hu et al. found that while images might show prohibited substances, profile bios and comment sections contain the explicit calls to action and contact methods that distinguish commercial intent from casual use. In medical misinformation tracking, Zhaozheng Yin et al. noted that while images provide context, the core deceptive claims are explicitly formulated in the text captions.
When evaluating the veracity of news, the linguistic structure remains the strongest predictor of truth. zhao kang et al. demonstrated that removing textual features from fake news detection frameworks causes a far more severe drop in performance than removing visual or emotional features. This is why lightweight, text-only models can perform nearly on par with massive multimodal systems for upstream triage tasks like checkworthiness detection (M. van der Meer et al.). The linguistic framing of a claim contains the primary markers of its verifiability.
Cross-Platform Robustness and the Invariance of Language
A major challenge in deep learning is generalization: models trained on one platform often fail when deployed on another. Multimodal models are particularly susceptible to this because social interaction graphs, propagation networks, and visual styles vary drastically between platforms.
Yupeng Li et al. exposed this vulnerability using their MCFEND benchmark, showing that complex multimodal and propagation-based fake news detectors suffer massive performance drops when moved across different Chinese social media platforms. In contrast, simple text-only models demonstrate superior cross-platform robustness. Deceptive language patterns remain relatively invariant across the web, whereas platform-specific features (like share counts or image compression styles) act as statistical traps that cause complex models to overfit.
This aligns with findings by Farig Sadeque et al. in online comment moderation, where adding social metadata failed to improve incivility detection. Identifying disrespectful behavior is a purely linguistic task independent of external social signals.
Transforming Visual Tasks into Textual Logic
The dominance of text is so pronounced that some of the most effective strategies for purely visual or multimodal tasks involve explicitly converting visual data into text before processing.
In advertisement understanding, Kanika Kalra et al. bypassed complex multimodal architectures entirely. By extracting slogans via scene-text OCR and generating dense image captions, they transformed the visual ad into a pure natural language string. This allowed a fine-tuned BERT model to achieve state-of-the-art accuracy, because the persuasive intent of an ad is heavily concentrated in its textual messaging.
| Approach | Input Modalities | Key Mechanism |
|---|---|---|
| Traditional Multimodal | Raw Pixels + Text Tokens | Early/Late Feature Fusion |
| Textual Transformation | OCR Text + Generated Captions | Pure NLP (e.g., BERT) |
Even in Scene Text Visual Question Answering (ST-VQA), the visual context often plays a secondary role. A study by Rubèn Tito revealed that humans can answer nearly twice as many questions using only the extracted text tokens compared to seeing the image with the text obscured. Similarly, when navigating complex scientific papers containing figures, tables, and equations, Anirudh Sundar et al. found that models sometimes achieve higher accuracy by processing the textual logic of the query and surrounding text alone, as visual components often serve as supplementary evidence rather than the primary locus of the argument.
This pattern holds true for clickbait spoiler identification. Vijayasaradhi Indurthi et al. and Ronghao Pan et al. found that features derived solely from the clickbait post's text and title are far more predictive of the spoiler type than the actual content of the linked document. The linguistic framing of the curiosity gap itself contains the strongest signal for classification, rendering external high-bandwidth document data largely redundant (Dragos-stefan Mihalcea et al.).
Why It Works
Textual information dominance is not a failure of multimodal modeling, but a reflection of how information is structured by humans. Language is an evolutionary compression mechanism designed specifically to convey discrete propositions, intents, and logical relationships.
Pixels, social graphs, and engagement metrics are high-dimensional, noisy, and highly context-dependent. When a model attempts to fuse these modalities, it often struggles with the lopsided signal-to-noise ratios. By prioritizing or isolating the textual signal, deep learning models leverage a highly dense, robust, and invariant representation of human thought that generalizes far better than the raw sensory data accompanying it.
Key Foundational Work
I2R at ImageCLEF Photo Retrieval 2009 (2009) — contributed 1 concept elements to this direction.
Fudan University at TRECVID 2006 (2006) — contributed 1 concept elements to this direction.