Introduction
Multimodal classification, which integrates diverse data types like images and text, typically yields superior performance compared to unimodal approaches. However, the efficacy of these models often hinges on the availability of vast human-annotated datasets. Semi-Supervised Multimodal Classification (SSMC) addresses this by leveraging a small set of labeled data alongside a larger pool of unlabeled data, primarily through pseudo-labeling.
Despite its promise, current SSMC methods face two significant challenges:
- Modality Interference: In standard fusion methods like Score Fusion, a noisy or incorrect prediction from one modality can degrade the overall accuracy, even if the other modality is correct.
- The Quality-Quantity Trade-off: Different fusion strategies exhibit varying behaviors; some produce high-quality but few pseudo-labels, while others generate many labels with lower accuracy.
The MSC Framework
The Modal and Strategic Complementarity (MSC) framework is designed to overcome these hurdles by introducing two core learning modules into standard semi-supervised backbones like FixMatch or FreeMatch.
Modal Complementarity Learning
To mitigate modality interference, MSC employs a Modal Reliability Generator. This sub-network dynamically assesses the trustworthiness of each modality for every specific instance. By assigning reliability weights, the model can effectively "mute" a confused modality, ensuring that confident predictions drive the final classification rather than being polluted by noise.
Strategic Complementarity Learning
MSC maintains two parallel fusion paths: Score Fusion (label-level) and Feature Concatenation (feature-level). Observations show that Score Fusion is often "picky" (high quality, low quantity), whereas Feature Concatenation is "greedy" (low quality, high quantity). The Strategic Complementarity Guidance (SCG) module uses dual KL divergence losses to facilitate mutual learning between these strategies, forcing the "greedy" path to improve its quality and the "picky" path to expand its coverage.
Experimental Evaluation
The framework was evaluated on three diverse benchmarks: N24News, UPMC-Food101, and CrisisMMD, simulating low-label scenarios with as few as 20 labels per class.
Key Results
- Superior Performance in Data Scarcity: On UPMC-Food101 with only 20 labels per class, MSC outperformed the strongest baseline by approximately 4-5% in accuracy.
- Competitive with Supervised Models: Remarkably, MSC achieved results comparable to state-of-the-art fully supervised models (such as UniS-MMC or MMBT) while using significantly less labeled data.
- Robust Integration: The framework proved effective when integrated with different semi-supervised backbones, demonstrating its generalizability.
Discussion and Insights
The success of MSC underscores the importance of dynamic weighting. Treating modality reliability as a per-instance learnable parameter allows the model to handle real-world data where one modality may be ambiguous. Furthermore, treating different fusion architectures as complementary "teachers and students" allows for internal knowledge distillation that offsets individual architectural weaknesses.
Limitations and Future Work
While MSC shows strong results for image-text pairs, future work may address the framework's scalability to more than two modalities (e.g., video or audio). Additionally, the computational overhead of maintaining dual fusion branches and the sensitivity to hyperparameter tuning for loss balancing remain areas for optimization.
Summary updated