Introduction
Multimodal Sentiment Analysis (MSA) and Emotion Recognition in Conversations (ERC) are essential for developing "empathetic" AI. However, integrating these capabilities into Large Language Models (LLMs) typically faces two hurdles: catastrophic forgetting, where fine-tuning degrades the model's general reasoning, and prohibitive computational costs associated with updating billions of parameters.
The MSE-Adapter addresses these challenges as a lightweight, "plug-and-play" plugin. By adding only ~2.7M trainable parameters to a frozen 7B model—an overhead of roughly 0.04%—it enables state-of-the-art multimodal performance while preserving the base LLM's original capabilities.
Methods: The MSE-Adapter Architecture
The MSE-Adapter sits between raw multimodal inputs and a frozen LLM, transforming vision and audio features into "pseudo tokens" that the text-based model can interpret.
The architecture relies on two primary components:
- Text-Guide-Mixer (TGM): This module uses a Hadamard product (element-wise multiplication) between textual features and non-textual (vision/audio) features. This alignment forces non-textual modalities into a feature space that is more "understandable" to the LLM.
- Multi-Scale Fusion (MSF): Recognizing that emotional cues occur at different granularities, MSF uses multiple MLPs with varying hidden dimensions to fuse vision and audio features across scales before projection.
The workflow involves extracting features from video and audio, modeling them temporally via sLSTM, aligning them using TGM, and finally fusing them through MSF into pseudo tokens. These tokens are concatenated with the text prompt and fed into the frozen LLM (e.g., LLaMA-2 or ChatGLM3).
Datasets and Evaluation
The model was evaluated across four diverse benchmarks to ensure cross-linguistic robustness:
- English: MOSEI (MSA) and MELD (ERC).
- Chinese: SIMS-V2 (MSA) and CHERMA (ERC).
Key Insights
- Alignment via Text-Guidance: The effectiveness of the TGM suggests that using text to "guide" other modalities allows the LLM to treat multimodal inputs as a familiar pattern, even if it wasn't originally trained on them.
- Extreme Efficiency: The minimal parameter count (2.6M–2.8M) allows training and deployment on consumer-grade hardware, such as a single NVIDIA RTX 4090.
- Pattern Matching: The authors noted that even when an LLM has limited native support for a language (e.g., LLaMA-2 with Chinese), the adapter facilitates successful mapping of multimodal patterns to sentiment labels.
Key Results
The MSE-ChatGLM3-6B variant achieved state-of-the-art (SOTA) results on most metrics. On the Chinese CHERMA dataset, it improved the Weighted F1 score by 2.19% over the previous best baseline.
Crucially, MSE-Adapter outperformed existing SOTA methods adapted to the same LLM backbones (like MAG and CHFN) while using significantly fewer trainable parameters. As shown in the comparison above, it achieved better performance than MAG while using only 2.63M parameters compared to MAG's 34.47M.
Limitations and Future Work
Performance remains heavily dependent on the "intelligence" of the base LLM; for instance, LLaMA-2's poor native Chinese understanding limited its performance on Chinese datasets regardless of the adapter. Future research may explore end-to-end training of the feature encoders and scaling the adapter logic to larger "frontier" models like Llama-3 70B.
Summary updated