When combining text with continuous modalities like audio and video, models face a fundamental asymmetry. Text is discrete, low-frequency, and highly dense in semantic meaning; a single word carries a precise concept. In contrast, audio and video streams are continuous, high-frequency, and semantically diffuse; a 25-millisecond audio frame contains raw acoustic pressure but almost no standalone semantic value.
Traditional multimodal fusion often relies on simple feature concatenation or late fusion. However, these approaches struggle because they force a network to discover complex latent alignments across vastly different granularities and noise levels. To solve this, a powerful paradigm has emerged: text-centric cross-modal attention fusion. Instead of treating all modalities symmetrically, this approach anchors continuous sensory streams directly to the discrete semantic space of language.
Text as the Semantic Query Anchor
The core intuition behind text-centric fusion lies in how the cross-attention mechanism assigns roles to different inputs. In a standard cross-attention layer, we define Queries ($Q$), Keys ($K$), and Values ($V$). By setting the textual features as the Query and the audio or visual features as the Key and Value, we force the model to look at continuous sensory streams through the lens of language.
Text Tokens (Query) ---------\
+---> Cross-Attention ---> Text-Grounded Multimodal Space
Audio/Video Frames (Key/Value) --/
When text acts as the query, the model asks: "Where in this continuous audio or video stream is the evidence for this specific linguistic concept?" This formulation allows non-textual modalities to modulate linguistic meaning effectively. As shown by Chudasama et al. in M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation, projecting non-verbal signals directly into the textual semantic space prevents the model from getting lost in acoustic noise, using the linguistic content as the primary framework for emotional context.
Similarly, Hou et al. demonstrated in Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach that text queries allow a model to selectively integrate features from audio and visual modalities. This prioritizes semantic content while leveraging non-verbal cues (like tone or gestures) to provide context for the primary textual information.
In tasks like toxicity detection, Maity et al. utilized this cross-attention mechanism in ToxVidLM: A Multimodal Framework for Toxicity Detection in Code-Mixed Videos to ensure that auxiliary visual and auditory features remain contextually relevant to the spoken utterance. This anchoring effect is also critical for long-form content; Wu et al. showed in Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmark that treating the textual narrative as a primary anchor allows the model to interpret and weight other sensory modalities effectively across long user-generated videos. Furthermore, Li et al. applied this in Traits Run Deep: Enhancing Personality Assessment via Psychology-Guided LLM Representations and Multimodal Apparent Behaviors, allowing auxiliary modalities to attend to text features to capture deeper behavioral clues.
Bridging the Granularity Gap
Even if text acts as the anchor, the model still faces a structural hurdle: a single text token (like a word or phoneme) spans dozens or hundreds of acoustic frames. How do we align these mismatched time scales?
One approach is to compress the continuous signal to match the linguistic scale. In Bridging the Granularity Gap for Acoustic Modeling, Xu et al. introduced Progressive Down-Sampling (PDS). By gradually compressing audio features across multiple stages, the acoustic encoder can distribute attention weights over semantically complete segments rather than fragmented audio frames.
Another strategy is to introduce explicit temporal supervision. For example, Bai et al. proposed A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing, which uses forced alignment during preprocessing to explicitly map phoneme sequences to corresponding acoustic frames. By assigning shared identifiers to phonemes and their associated audio frames, the cross-modal attention mechanism can prioritize high-fidelity signal reconstruction over the discovery of latent alignments during training.
Similarly, Yu et al. developed SPECTRA in Speech-Text Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment, supervising the modality fusion module with explicit word-level timestamps to dynamically bind speech and text representations during multi-turn conversations.
Alternatively, this alignment can be learned end-to-end without explicit timestamps. In Multimodal Semi-Supervised Learning Framework for Punctuation Prediction in Conversational Speech, Sunkara et al. demonstrated that a scaled dot-product attention mechanism can automatically align audio frames with text sub-words, dynamically weighting relevant acoustic features for each text token end-to-end.
Learning Temporal Correlations and Agreement
Once text and audio are aligned in a shared latent space, cross-attention matrices can be used to measure the precise agreement between modalities. This is highly effective for open-vocabulary tasks and temporal boundary detection.
In Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting, Shin et al. used cross-attention to generate an attention matrix capturing nuanced temporal correlations between audio and text embeddings. This allows the model to spot user-defined keywords in noisy environments without requiring prior audio enrollment. For long-form audio, Wu et al. introduced CoLLAP in Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation, which uses multimodal temporal attention with kernel-wise weights to map specific segments of a 5-minute audio stream to corresponding sections of a detailed text description.
This fine-grained temporal tracking also enables models to translate acoustic properties back into natural language or structural labels. For instance, Guo et al. developed SED-LM in Leveraging Language Model Capabilities for Sound Event Detection, pairing acoustic encoders with large language models via cross-attention. By treating audio features as keys/values and text tokens as queries, the architecture aligns fine-grained temporal audio data with semantic category descriptions, allowing the model to output exactly when a sound occurs using natural language templates. In a similar vein, Dai et al. showed in Automatic Prosody Annotation with Pre-Trained Text-Speech Model that using text-based contextual embeddings as queries allows a decoder to identify specific temporal boundaries in audio corresponding to linguistic transitions.
Overcoming Linguistic and Acoustic Ambiguity
Finally, text-anchored alignment makes models highly resilient to errors in individual modalities. In spoken language processing, speech recognition errors or acoustic noise can degrade performance. By learning a joint representation, the modalities can mutually disambiguate each other.
You et al. demonstrated in Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering that cross-modal temporal alignment ensures models associate specific time-aligned audio signals with their corresponding linguistic meanings, making spoken question answering systems significantly more resilient to speech recognition errors.
Conversely, acoustic cues can resolve textual ambiguities. In A text-speech multimodal Chinese named entity recognition model for crop diseases and pests, Liu et al. showed that aligning text queries with audio keys enables a model to leverage natural pauses and phonemic cues in spoken language. This resolves ambiguous word boundaries in text, yielding superior performance over traditional text-only methods.
Key Foundational Work
Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering (2021) — contributed 1 concept elements to this direction.
Multimodal Semi-Supervised Learning Framework for Punctuation Prediction in Conversational Speech (2020) — contributed 1 concept elements to this direction.