Traditional multimodal models attempt to map raw sensory features—such as image pixels, audio waveforms, and video frames—directly into a shared embedding space or a joint text generation head. While powerful, training these end-to-end architectures requires massive computational resources and often struggles with fine-grained logical reasoning, contextual nuances, and resolving redundancies across different sensory streams.
A compelling alternative paradigm has emerged: using text-only or frozen Large Language Models (LLMs) as central multimodal fusion engines. Instead of fusing raw, high-dimensional features, this approach uses specialized "expert" models to translate individual sensory modalities into discrete textual descriptions. The LLM then acts as a reasoning bridge, synthesizing these fragmented, modality-specific annotations into a unified, coherent, and contextually rich narrative.
Text as a Universal Intermediate Representation
Why does this text-mediated fusion work so well? Raw sensory data is inherently noisy and high-dimensional. Aligning an audio clip of a dog barking with a video frame of a dog running requires complex cross-attention mechanisms that must learn both spatial and temporal alignment from scratch.
Text, however, strips away sensory noise while retaining core semantic meaning. By converting each modality into intermediate text snippets (e.g., via automatic speech recognition, optical character recognition, or object detection), the problem of multimodal fusion is transformed into a text-to-text synthesis problem. This allows researchers to offload the complex task of cross-modal reasoning to the highly developed, pre-trained semantic spaces of LLMs.
[Raw Video] ───> (Vision Expert) ───> "A dog runs in a park" ──┐
[Raw Audio] ───> (Audio Expert) ───> [Barking sounds] ──┼─> [ Frozen LLM ] ───> Enriched, Coherent Caption
[Subtitles] ───> (Parser) ───> "Come here, boy!" ──┘
Rather than simply concatenating these text snippets—which results in repetitive, cluttered, and disjointed prompts—the LLM is prompted to act as an intelligent synthesis engine. As Shunian Chen et al. demonstrate in Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion, an LLM can move beyond simple aggregation to resolve redundancies, infer hidden context, and generate highly detailed, fine-grained descriptions from diverse audio, speech, music, and video cues.
Resolving Redundancy and Aligning Context
When multiple specialized models describe the same scene, their outputs inevitably overlap. An object detector might output "car," while an image captioner outputs "a vehicle on the road."
LLMs excel at filtering out this semantic redundancy. In Zhoumou at SemEval-2025 Task 1: Leveraging Multimodal Data Augmentation and Large Language Models for Enhanced Idiom Understanding, Yingzhou Zhao et al. employ an LLM specifically to condense original and generated text descriptions into a concise, semantically rich representation, eliminating clutter while preserving critical information. Similarly, Yiguo He et al. show in Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation that LLMs can effectively consolidate multiple diverse image descriptions, filtering out non-visual information and ensuring strict image-text alignment.
This capability is also vital for structuring unstructured data across disparate formats. Nazarii Drushchak et al. leverage this in Multimodal Retrieval-Augmented Generation: Unified Information Processing Across Text, Image, Table, and Video Modalities, using LLMs to extract semantic meaning, summarize tables, and generate metadata for video keyframes to enable unified processing across four distinct modalities.
Bridging Sparse Labels to Rich Narratives
A major bottleneck in training generative models (such as text-to-image or text-to-audio generators) is the scarcity of high-quality, descriptive training prompts. Most datasets contain only sparse labels or generic captions. LLM-driven synthesis provides an automated pipeline to enrich these datasets at scale.
For instance, Noam Rotstein et al. introduced FuseCap in Leveraging Large Language Models for Enriched Fused Image Captions, a data-centric framework that automatically enriches generic image captions by fusing them with detailed information from specialized vision experts (object detectors and OCR) via an LLM. This pipeline transforms disparate, category-specific labels into a unified, natural language narrative.
This enrichment process is particularly useful for resolving semantic ambiguities or sensory illusions in generative tasks:
- Audio-to-Image Generation: Hyunwoo Oh et al. developed CatchPhrase in CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation, which synthesizes multi-faceted visual, auditory, and semantic contexts to transform ambiguous class labels into detailed prompts, preventing downstream generative models from falling victim to homographs or auditory illusions.
- Audible-Video Generation: In TAVGBench: Benchmarking Text to Audible-Video Generation, Xuyang Shen et al. utilize LLMs as linguistic bridges to merge separate modality-specific annotations into unified, high-quality training prompts, ensuring that a single description accurately captures both the visual and auditory components of a scene.
Synthesizing Multi-Sensory and Temporal Data
When applied to video, the LLM's role expands from a simple aggregator to a temporal and narrative reasoning engine. Video understanding requires capturing the intersection of what is seen, what is heard, and what is spoken over time.
In VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset, Sihan Chen et al. use an LLM to synthesize vision captions, audio captions, and raw subtitles into a unified linguistic representation of multi-sensory data. Similarly, Yi Yuan et al. developed an automated pipeline in Sound-VECaps: Improving Audio Generation With Visual Enhanced Captions that integrates visual information from multiple video frames with concise audio captions and labels, leveraging the LLM to preserve the correct temporal ordering of events.
This synthesis capability can also be used to transfer reasoning across modalities or compress complex educational content:
- Reasoning Transfer: Edson Araujo et al. present AVRT in AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers, demonstrating that a text-only LLM can effectively serve as a merger to combine independent reasoning outputs from different single-modality teachers into a unified, standardized reasoning trace.
- Contextual Summarization: Kazuki Kawamura et al. introduce FastPerson in FastPerson: Enhancing Video-Based Learning through Video Summarization that Preserves Linguistic and Visual Contexts, using an LLM to synthesize speech-to-text transcripts, slide OCR, and object detection labels into a unified lecture summary that preserves the instructor's original voice and context.
- Narrative Generation: For complex media like film, Xiaojun Ye et al. show in MMAD: Multi-modal Movie Audio Description that aligning video, ambient sound, and character data into an LLM's feature space allows the model to leverage its internal reasoning capabilities to synthesize complex narrative elements for the visually impaired.
Injecting Domain-Specific Knowledge
A final advantage of using LLMs as fusion engines is their ability to seamlessly incorporate domain-specific constraints, cultural context, or expert knowledge during the synthesis process.
In cultural contexts, standard vision models often mislabel region-specific items. To fix this, Youngsik Yun developed the CIC framework in Culturally-aware Image Captioning, which uses targeted Visual Question Answering (VQA) to extract cultural details and an LLM to synthesize those attributes with a base caption into a contextually accurate description.
In highly technical domains like medicine, LLMs bridge the gap between raw quantitative metrics and clinical narratives:
- Radiology Reports: The authors of RadGPT: Constructing 3D Image-Text Tumor Datasets integrate 3D segmentation outputs with an LLM. By extracting quantitative attributes from tumor segmentations to build structured data, they use the LLM to process and transform these metrics into stylized, narrative radiology reports.
- Cardiovascular Interpretations: In MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretations, Deyun Zhang et al. employ LLMs to transform sparse clinical notes and raw quantitative ECG metrics into detailed, structured narratives. By conditioning the text generation on both expert observations and numerical parameters, the LLM successfully explains the logical relationship between raw data and final diagnosis.
Open Challenges
While using LLMs as multimodal fusion engines bypasses the need for expensive end-to-end cross-modal training, it introduces a new challenge: prompt topology.
As noted by Junting Pan et al. in Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models, determining the optimal linguistic structure for integrating retrieved visual descriptions into a language model's prompt remains an open problem. Future work must continue to explore how different natural language markers and prompt configurations affect the model's ability to accurately synthesize disparate textual snippets into a coherent temporal narrative without introducing hallucinations.
Key Foundational Work
Sound-VECaps: Improving Audio Generation With Visual Enhanced Captions (2024) — contributed 3 concept elements to this direction.
Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning (2024) — contributed 3 concept elements to this direction.
TAVGBench: Benchmarking Text to Audible-Video Generation (2024) — contributed 2 concept elements to this direction.