In multimodal research—where models process text, audio, and video simultaneously—there is a tempting intuition that all data streams should be treated as equal partners. However, recent evidence suggests that human sentiment and intent are not distributed evenly across these channels. Instead, a "modality hierarchy" exists where textual data acts as a primary semantic anchor, while audio and visual cues serve as secondary modifiers.
The Equality Fallacy and Textual Dominance
Early approaches to multimodal fusion often employed symmetric architectures, assuming that the model should learn to bridge the gap between all modalities with equal weight. However, as noted by Devamanyu Hazarika et al. in MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis, this imbalance is partly driven by the availability of high-quality pre-trained linguistic models. Text provides a dense, abstract signal that is often more discriminative than the noisier, more ambiguous signals found in facial expressions or vocal tones.
Empirical evidence consistently places text at the top of the hierarchy. Anirudh Bindiganavale Harish et al. found that textual data contributes approximately half of the final decision in sentiment analysis, while Soujanya Poria et al. demonstrated that linguistic features alone provide the strongest baseline for understanding human sentiment. This dominance extends beyond sentiment; in video retrieval, Anastasia Moumtzidou et al. showed that speech transcripts provide a significantly stronger signal for identifying content than visual concept detection.
Text as the Semantic Anchor
If text is the most reliable signal, how should a model use the other modalities? The emerging consensus is to treat text as an anchor. In this framing, the text defines the core intent or polarity, while audio and video refine the intensity or resolve linguistic ambiguity.
Amir Zadeh et al. proposed in Tensor Fusion Network for Multimodal Sentiment Analysis that non-verbal cues function as modifiers that can shift the intensity or even reverse the polarity of the text (e.g., sarcasm). Similarly, Songning Lai et al. argue that while text provides the primary signal for emotional polarity, other modalities are essential for capturing nuances that single-modality systems miss.
To exploit this, researchers have moved toward "text-centered" frameworks. Yang Wu et al. developed a framework that explicitly separates non-textual information into shared semantics that reinforce the text and private semantics that provide unique clues. By treating text as the anchor, the model avoids the "dilution" of high-quality information by noisier auxiliary data, a problem identified by Zilong Huang et al. in their work on conversational emotion recognition.
Asymmetric Fusion Mechanisms
The shift toward text-centricity has led to several specific architectural mechanisms:
- Cross-Modal Distillation: Rather than just fusing features, models can "distill" knowledge from a powerful text model into audio or visual "student" models. Ruichen Li et al. used this to improve speech emotion recognition, while Taeyang Yun et al. employed a "teacher-leading" network where non-verbal cues are used specifically to refine textual representations.
- Hierarchical Denoising: Text can act as a guide to filter noise from other channels. Zuocheng Li et al. introduced t-HNE, which uses text to hierarchically denoise visual and audio data, assuming that the textual sentiment is a more stable ground truth.
- Biased Interaction: Instead of uniform attention, models like those proposed by Haozhe Yang et al. use biased interaction strategies that prioritize text-based features, ensuring that the primary anchor is never lost in the fusion process.
Robustness and Human Perception
This hierarchy is not just a computational shortcut; it reflects how humans perceive emotion. Quanqi Du et al. found that textual information exerts a disproportionately high influence on human annotators when they assign emotion labels.
The practical consequence of this hierarchy is most visible when data is missing. Yuanzhi Wang et al. demonstrated that in incomplete multimodal datasets, prioritizing the recovery of missing text features yields significantly higher performance gains than recovering acoustic or visual features. As Pan Wang et al. argue in their work on disentangled-language-focused models, the future of the field lies not in treating all inputs as equal, but in strategically enhancing the dominant signals that anchor human communication.