Abstract
This study investigates whether Large Language Models (LLMs) trained on multimodal data (specifically audio) or human-to-human dialogue can transfer social intuition to text-only tasks. By comparing models like GPT-4o and conversationally-tuned Llama-2 variants, the researchers found that multimodal training significantly enhances the detection of "covert" deception—such as sarcasm, irony, and condescension—in pure text environments. These findings suggest that social features learned from speech prosody and interactional structure can successfully transfer across modalities.
Introduction
Human communication is rarely literal. We frequently employ covert deception, where the connotation (intended meaning) is the polar opposite of the denotation (literal words). For instance, a sarcastic "That’s just great!" following a mishap relies on acoustic cues like rhythm, stress, and intonation—collectively known as prosody.
While humans develop an "ear" for these nuances in childhood and eventually transfer that intuition to written text, standard LLMs are primarily trained on massive text crawls that lack these acoustic signals. This paper asks: If an LLM is trained on audio (like GPT-4o) or specifically on human-to-human dialogue, does it develop a superior "social intuition" for detecting subtle linguistic nuances in text-only environments?
Methodology and Datasets
The researchers conducted a "head-to-head" comparison between two pairs of models to isolate the impact of training data:
- The Multimodal Pair: GPT-4o (trained on text, image, and audio) vs. GPT-4-Turbo (trained on text and image).
- The Conversation Pair: Llama-2-70B-chat (standard internet text) vs. Llama-2-70B-conversational (fine-tuned specifically on human-to-human dialogue).
Models were tested across four prompting strategies: zero-shot, few-shot, and both with and without Chain-of-Thought (CoT) reasoning. The study utilized high-quality subcomponents from the Big-Bench benchmark, including Snarks (sarcasm), Irony Identification, and TalkDown (condescension).
Key Insights and Results
Implicit Multimodal Transfer
The results indicate that implicit multimodal transfer is a real phenomenon. GPT-4o, which was exposed to speech during training, outperformed GPT-4-Turbo on text-only sarcasm and irony detection. On average, GPT-4o showed a 2.2% advantage in accuracy and a 2.6% advantage in F1-score over its less-multimodal counterpart when using basic prompts.
The "Intuition vs. Reasoning" Gap
Interestingly, GPT-4o’s advantage was most pronounced in "out-of-the-box" (non-CoT) scenarios. When forced to use Chain-of-Thought reasoning, the performance gap often closed or flipped. This implies that multimodal training provides a better intuitive grasp of social cues, whereas text-only models may need to "think it through" analytically to reach similar conclusions.
Conversational Tuning as a Proxy
Fine-tuning on human dialogue (Llama-2-conversational) provided similar boosts to social nuance detection. For condescension detection, the conversationally-tuned model saw a 14.1% boost in accuracy and a 17.4% boost in precision over the standard chat model.
Discussion and Limitations
While the results are promising, the authors note several limitations. Because GPT-4o is closed-source, it is difficult to determine if its edge stems strictly from audio data or a larger text corpus. Additionally, some datasets (like Irony) are relatively small, leading to higher variance in results.
Furthermore, prompting strategies intended to help—such as telling a model to "think about how this sounds"—actually hurt performance on average. This suggests that social features are deeply integrated into the model's internal representations; forcing explicit focus on them via prompting may introduce unnecessary noise.
Conclusion
This research demonstrates that LLMs can indeed learn to "read between the lines" through multimodal exposure. Future work will explore trained approaches that explicitly leverage multimodal features to improve text-only performance and investigate how these models handle complex, context-dependent social interactions beyond single-sentence snippets.
Summary updated