Large Language Models as Multimodal Fusion Engines

25 concepts17 papers1 proposal

Traditional multimodal models attempt to map raw sensory features—such as image pixels, audio waveforms, and video frames—directly into a shared embedding space or a joint text generation head. While powerful, training these end-to-end architectures requires massive computational resources and often struggles with fine-grained logical reasoning, contextual nuances, and resolving redundancies across different sensory streams.

A compelling alternative paradigm has emerged: using text-only or frozen Large Language Models (LLMs) as central multimodal fusion engines. Instead of fusing raw, high-dimensional features, this approach uses specialized "expert" models to translate individual sensory modalities into discrete textual descriptions. The LLM then acts as a reasoning bridge, synthesizing these fragmented, modality-specific annotations into a unified, coherent, and contextually rich narrative.

Text as a Universal Intermediate Representation

Why does this text-mediated fusion work so well? Raw sensory data is inherently noisy and high-dimensional. Aligning an audio clip of a dog barking with a video frame of a dog running requires complex cross-attention mechanisms that must learn both spatial and temporal alignment from scratch.

Text, however, strips away sensory noise while retaining core semantic meaning. By converting each modality into intermediate text snippets (e.g., via automatic speech recognition, optical character recognition, or object detection), the problem of multimodal fusion is transformed into a text-to-text synthesis problem. This allows researchers to offload the complex task of cross-modal reasoning to the highly developed, pre-trained semantic spaces of LLMs.

[Raw Video]  ───> (Vision Expert) ───> "A dog runs in a park" ──┐
[Raw Audio]  ───> (Audio Expert)  ───> [Barking sounds]       ──┼─> [ Frozen LLM ] ───> Enriched, Coherent Caption
[Subtitles]  ───> (Parser)        ───> "Come here, boy!"      ──┘

Rather than simply concatenating these text snippets—which results in repetitive, cluttered, and disjointed prompts—the LLM is prompted to act as an intelligent synthesis engine. As Shunian Chen et al. demonstrate in Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion, an LLM can move beyond simple aggregation to resolve redundancies, infer hidden context, and generate highly detailed, fine-grained descriptions from diverse audio, speech, music, and video cues.

Resolving Redundancy and Aligning Context

When multiple specialized models describe the same scene, their outputs inevitably overlap. An object detector might output "car," while an image captioner outputs "a vehicle on the road."

LLMs excel at filtering out this semantic redundancy. In Zhoumou at SemEval-2025 Task 1: Leveraging Multimodal Data Augmentation and Large Language Models for Enhanced Idiom Understanding, Yingzhou Zhao et al. employ an LLM specifically to condense original and generated text descriptions into a concise, semantically rich representation, eliminating clutter while preserving critical information. Similarly, Yiguo He et al. show in Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation that LLMs can effectively consolidate multiple diverse image descriptions, filtering out non-visual information and ensuring strict image-text alignment.

This capability is also vital for structuring unstructured data across disparate formats. Nazarii Drushchak et al. leverage this in Multimodal Retrieval-Augmented Generation: Unified Information Processing Across Text, Image, Table, and Video Modalities, using LLMs to extract semantic meaning, summarize tables, and generate metadata for video keyframes to enable unified processing across four distinct modalities.

Bridging Sparse Labels to Rich Narratives

A major bottleneck in training generative models (such as text-to-image or text-to-audio generators) is the scarcity of high-quality, descriptive training prompts. Most datasets contain only sparse labels or generic captions. LLM-driven synthesis provides an automated pipeline to enrich these datasets at scale.

For instance, Noam Rotstein et al. introduced FuseCap in Leveraging Large Language Models for Enriched Fused Image Captions, a data-centric framework that automatically enriches generic image captions by fusing them with detailed information from specialized vision experts (object detectors and OCR) via an LLM. This pipeline transforms disparate, category-specific labels into a unified, natural language narrative.

This enrichment process is particularly useful for resolving semantic ambiguities or sensory illusions in generative tasks:

Synthesizing Multi-Sensory and Temporal Data

When applied to video, the LLM's role expands from a simple aggregator to a temporal and narrative reasoning engine. Video understanding requires capturing the intersection of what is seen, what is heard, and what is spoken over time.

In VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset, Sihan Chen et al. use an LLM to synthesize vision captions, audio captions, and raw subtitles into a unified linguistic representation of multi-sensory data. Similarly, Yi Yuan et al. developed an automated pipeline in Sound-VECaps: Improving Audio Generation With Visual Enhanced Captions that integrates visual information from multiple video frames with concise audio captions and labels, leveraging the LLM to preserve the correct temporal ordering of events.

This synthesis capability can also be used to transfer reasoning across modalities or compress complex educational content:

Injecting Domain-Specific Knowledge

A final advantage of using LLMs as fusion engines is their ability to seamlessly incorporate domain-specific constraints, cultural context, or expert knowledge during the synthesis process.

In cultural contexts, standard vision models often mislabel region-specific items. To fix this, Youngsik Yun developed the CIC framework in Culturally-aware Image Captioning, which uses targeted Visual Question Answering (VQA) to extract cultural details and an LLM to synthesize those attributes with a base caption into a contextually accurate description.

In highly technical domains like medicine, LLMs bridge the gap between raw quantitative metrics and clinical narratives:

  • Radiology Reports: The authors of RadGPT: Constructing 3D Image-Text Tumor Datasets integrate 3D segmentation outputs with an LLM. By extracting quantitative attributes from tumor segmentations to build structured data, they use the LLM to process and transform these metrics into stylized, narrative radiology reports.
  • Cardiovascular Interpretations: In MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretations, Deyun Zhang et al. employ LLMs to transform sparse clinical notes and raw quantitative ECG metrics into detailed, structured narratives. By conditioning the text generation on both expert observations and numerical parameters, the LLM successfully explains the logical relationship between raw data and final diagnosis.

Open Challenges

While using LLMs as multimodal fusion engines bypasses the need for expensive end-to-end cross-modal training, it introduces a new challenge: prompt topology.

As noted by Junting Pan et al. in Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models, determining the optimal linguistic structure for integrating retrieved visual descriptions into a language model's prompt remains an open problem. Future work must continue to explore how different natural language markers and prompt configurations affect the model's ability to accurately synthesize disparate textual snippets into a coherent temporal narrative without introducing hallucinations.

Key Foundational Work

Sound-VECaps: Improving Audio Generation With Visual Enhanced Captions (2024) — contributed 3 concept elements to this direction.
Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning (2024) — contributed 3 concept elements to this direction.
TAVGBench: Benchmarking Text to Audible-Video Generation (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multimodal_caption_synthesis_7574f164,
  title = {multimodal caption synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-caption-synthesis/rdr_7574f164d0784428b377a25ea4fd7008},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_multimodal_caption_synthesis_7574f164_v1,
  title = {multimodal caption synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-caption-synthesis/rdr_7574f164d0784428b377a25ea4fd7008/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake