Understanding Multimodal Learning: A Loss Landscape Smoothness Perspective
Lists
Only you see this
Abstract
A surge of recent advancements has consistently highlighted the superiority of multimodal learning over unimodal approaches across a variety of tasks. However, the theoretical foundations elucidating this advantage remain underexplored: existing theoretical analyses are often constrained by tight assumptions, and lack empirical validation. In this paper, we bridge this gap by proposing a novel theoretical framework grounded in convolutional smoothing, offering a new perspective on how multimodal learning contributes to a smoother loss landscape compared to unimodal learning. Building upon this theoretical foundation, we introduce a simple yet effective distributional training strategy based on stochastic modality pairing instead of a fixed pairing; thus, further promoting a flatter landscape via convolutional smoothing. Our empirical results across various multimodal datasets demonstrate that multimodal models not only achieve higher performance but also exhibit flatter loss landscape, which represent better generalization and robustness.
BibTeX
@inproceedings{lee2026understanding,
author = {Jae-Jun Lee and Sung Whan Yoon},
title = {{Understanding Multimodal Learning: A Loss Landscape Smoothness Perspective}},
booktitle = {International Conference on Machine Learning},
year = {2026},
url = {https://openreview.net/forum?id=2htwc6cHmu},
}Loading related content...