"Yeah Right!" - Do LLMs Exhibit Multimodal Feature Transfer?

Benjamin Z. Reichman, Kartik TalamadupulaarXiv 2025 · DOI · arXiv

Abstract

This study investigates whether Large Language Models (LLMs) trained on multimodal data (specifically audio) or human-to-human dialogue can transfer social intuition to text-only tasks. By comparing models like GPT-4o and conversationally-tuned Llama-2 variants, the researchers found that multimodal training significantly enhances the detection of "covert" deception—such as sarcasm, irony, and condescension—in pure text environments. These findings suggest that social features learned from speech prosody and interactional structure can successfully transfer across modalities.

Introduction

Human communication is rarely literal. We frequently employ covert deception, where the connotation (intended meaning) is the polar opposite of the denotation (literal words). For instance, a sarcastic "That’s just great!" following a mishap relies on acoustic cues like rhythm, stress, and intonation—collectively known as prosody.

While humans develop an "ear" for these nuances in childhood and eventually transfer that intuition to written text, standard LLMs are primarily trained on massive text crawls that lack these acoustic signals. This paper asks: If an LLM is trained on audio (like GPT-4o) or specifically on human-to-human dialogue, does it develop a superior "social intuition" for detecting subtle linguistic nuances in text-only environments?

Methodology and Datasets

The researchers conducted a "head-to-head" comparison between two pairs of models to isolate the impact of training data:

  1. The Multimodal Pair: GPT-4o (trained on text, image, and audio) vs. GPT-4-Turbo (trained on text and image).
  2. The Conversation Pair: Llama-2-70B-chat (standard internet text) vs. Llama-2-70B-conversational (fine-tuned specifically on human-to-human dialogue).

Models were tested across four prompting strategies: zero-shot, few-shot, and both with and without Chain-of-Thought (CoT) reasoning. The study utilized high-quality subcomponents from the Big-Bench benchmark, including Snarks (sarcasm), Irony Identification, and TalkDown (condescension).

Table 1: Examples of covert deception types used in the study datasets, showing the contrast between positive and negative cases.

Key Insights and Results

Implicit Multimodal Transfer

The results indicate that implicit multimodal transfer is a real phenomenon. GPT-4o, which was exposed to speech during training, outperformed GPT-4-Turbo on text-only sarcasm and irony detection. On average, GPT-4o showed a 2.2% advantage in accuracy and a 2.6% advantage in F1-score over its less-multimodal counterpart when using basic prompts.

The "Intuition vs. Reasoning" Gap

Interestingly, GPT-4o’s advantage was most pronounced in "out-of-the-box" (non-CoT) scenarios. When forced to use Chain-of-Thought reasoning, the performance gap often closed or flipped. This implies that multimodal training provides a better intuitive grasp of social cues, whereas text-only models may need to "think it through" analytically to reach similar conclusions.

Conversational Tuning as a Proxy

Fine-tuning on human dialogue (Llama-2-conversational) provided similar boosts to social nuance detection. For condescension detection, the conversationally-tuned model saw a 14.1% boost in accuracy and a 17.4% boost in precision over the standard chat model.

Table 2: Performance gains of Llama-2-conversational over Llama-2-chat across various datasets and reasoning modes.

Discussion and Limitations

While the results are promising, the authors note several limitations. Because GPT-4o is closed-source, it is difficult to determine if its edge stems strictly from audio data or a larger text corpus. Additionally, some datasets (like Irony) are relatively small, leading to higher variance in results.

Furthermore, prompting strategies intended to help—such as telling a model to "think about how this sounds"—actually hurt performance on average. This suggests that social features are deeply integrated into the model's internal representations; forcing explicit focus on them via prompting may introduce unnecessary noise.

Conclusion

This research demonstrates that LLMs can indeed learn to "read between the lines" through multimodal exposure. Future work will explore trained approaches that explicitly leverage multimodal features to improve text-only performance and investigate how these models handle complex, context-dependent social interactions beyond single-sentence snippets.

Summary updated

BibTeX
@misc{reichman2025yeah,
  author = {Benjamin Z. Reichman and Kartik Talamadupula},
  title = {{"Yeah Right!" - Do LLMs Exhibit Multimodal Feature Transfer?}},
  year = {2025},
  eprint = {2501.04138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.04138},
}
Figure