Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities

Junchi Chen, Richong Zhang, Junfan Chen

Introduction

Multimodal classification, which integrates diverse data types like images and text, typically yields superior performance compared to unimodal approaches. However, the efficacy of these models often hinges on the availability of vast human-annotated datasets. Semi-Supervised Multimodal Classification (SSMC) addresses this by leveraging a small set of labeled data alongside a larger pool of unlabeled data, primarily through pseudo-labeling.

Despite its promise, current SSMC methods face two significant challenges:

  1. Modality Interference: In standard fusion methods like Score Fusion, a noisy or incorrect prediction from one modality can degrade the overall accuracy, even if the other modality is correct.
  2. The Quality-Quantity Trade-off: Different fusion strategies exhibit varying behaviors; some produce high-quality but few pseudo-labels, while others generate many labels with lower accuracy.
Conceptual overview of Modal and Strategic Complementarity.

The MSC Framework

The Modal and Strategic Complementarity (MSC) framework is designed to overcome these hurdles by introducing two core learning modules into standard semi-supervised backbones like FixMatch or FreeMatch.

To mitigate modality interference, MSC employs a Modal Reliability Generator. This sub-network dynamically assesses the trustworthiness of each modality for every specific instance. By assigning reliability weights, the model can effectively "mute" a confused modality, ensuring that confident predictions drive the final classification rather than being polluted by noise.

Strategic Complementarity Learning

MSC maintains two parallel fusion paths: Score Fusion (label-level) and Feature Concatenation (feature-level). Observations show that Score Fusion is often "picky" (high quality, low quantity), whereas Feature Concatenation is "greedy" (low quality, high quantity). The Strategic Complementarity Guidance (SCG) module uses dual KL divergence losses to facilitate mutual learning between these strategies, forcing the "greedy" path to improve its quality and the "picky" path to expand its coverage.

The MSC architecture, featuring Modal Reliability Guidance and Strategic Complementarity Guidance.

Experimental Evaluation

The framework was evaluated on three diverse benchmarks: N24News, UPMC-Food101, and CrisisMMD, simulating low-label scenarios with as few as 20 labels per class.

Key Results

  • Superior Performance in Data Scarcity: On UPMC-Food101 with only 20 labels per class, MSC outperformed the strongest baseline by approximately 4-5% in accuracy.
  • Competitive with Supervised Models: Remarkably, MSC achieved results comparable to state-of-the-art fully supervised models (such as UniS-MMC or MMBT) while using significantly less labeled data.
  • Robust Integration: The framework proved effective when integrated with different semi-supervised backbones, demonstrating its generalizability.

Discussion and Insights

The success of MSC underscores the importance of dynamic weighting. Treating modality reliability as a per-instance learnable parameter allows the model to handle real-world data where one modality may be ambiguous. Furthermore, treating different fusion architectures as complementary "teachers and students" allows for internal knowledge distillation that offsets individual architectural weaknesses.

Limitations and Future Work

While MSC shows strong results for image-text pairs, future work may address the framework's scalability to more than two modalities (e.g., video or audio). Additionally, the computational overhead of maintaining dual fusion branches and the sensitivity to hyperparameter tuning for loss balancing remain areas for optimization.

Summary updated

Abstract

Supervised multimodal classification has been proven to outperform unimodal classification in the image-text domain. However, this task is highly dependent on abundant labeled data. To perform multimodal classification in data-insufficient scenarios, in this study, we explore semi-supervised multimodal classification (SSMC) that only requires a small amount of labeled data and plenty of unlabeled data. Specifically, we first design baseline SSMC models by combining known semi supervised pseudo-labeling methods with the two most commonly used modal fusion strategies, i.e. feature-level fusion and label-level aggregation. Based on our investigation and empirical study of the baselines, we discover two complementarities that may benefit SSMC if properly exploited: the predictions from different modalities (modal complementarity) and modal fusion strategies for pseudo-labeling (strategic complementarity). Therefore, we propose a Modal and Strategic Complementarity (MSC) framework for SSMC. Concretely, to exploit modal complementarity, we propose to learn reliability weights for the predictions from different modalities and refine the fusion scores. To learn from strategic complementarity, we introduce a dual KL divergence loss to guide the balance of quantity and quality of pseudo-labeled data selection. Extensive empirical studies demonstrate the effectiveness of the proposed framework.
BibTeX
@inproceedings{chen2025semi,
  author = {Junchi Chen and Richong Zhang and Junfan Chen},
  title = {{Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33736},
}
Figure