MuMu: Cooperative Multitask Learning-Based Guided Multimodal Fusion

Md Mofijul Islam, Tariq IqbalAAAI 2022 · OpenReview

Abstract

Multimodal sensors (visual, non-visual, and wearable) can provide complementary information to develop robust perception systems for recognizing activities accurately. However, it is challenging to extract robust multimodal representations due to the heterogeneous characteristics of data from multimodal sensors and disparate human activities, especially in the presence of noisy and misaligned sensor data. In this work, we propose a cooperative multitask learning-based guided multimodal fusion approach, MuMu, to extract robust multimodal representations for human activity recognition (HAR). MuMu employs an auxiliary task learning approach to extract features specific to each set of activities with shared characteristics (activity-group). MuMu then utilizes activity-group-specific features to direct our proposed Guided Multimodal Fusion Approach (GM-Fusion) for extracting complementary multimodal representations, designed as the target task. We evaluated MuMu by comparing its performance to state-of-the-art multimodal HAR approaches on three activity datasets. Our extensive experimental results suggest that MuMu outperforms all the evaluated approaches across all three datasets.

Introduction

Human Activity Recognition (HAR) is a cornerstone of modern assistive technologies, autonomous systems, and healthcare monitoring. By integrating diverse modalities—such as RGB video, depth maps, and inertial measurement unit (IMU) data—systems can achieve a more holistic understanding of human movement. However, two primary challenges persist: the heterogeneity of activities and the inherent "messiness" of real-world sensor data.

Different activities often rely on specific modalities; for instance, fine-grained hand movements are best captured by visual or wrist-worn sensors, while gross motor skills like running are more evident in leg-mounted IMUs. Furthermore, real-world deployments frequently encounter sensor noise, occlusions, and temporal misalignments. Standard fusion techniques often fail when a primary modality becomes unreliable.

Methodology: The MuMu Framework

MuMu addresses these challenges through a cooperative multitask learning architecture. The framework is designed to leverage high-level activity context to guide the fusion process.

MuMu Cooperative Multitask Learning Architecture

The architecture comprises three core components:

  1. Unimodal Feature Encoders (UFE): These encoders (e.g., ResNet-50 for video, LSTMs for time-series) extract independent feature representations from each sensor modality.
  2. Auxiliary Task Learning (ATL): This module learns to classify broad "activity groups" (e.g., desk-based vs. complex movements). The resulting feature vector ($X^{aux}$) provides a high-level context of the action.
  3. Guided Multimodal Fusion (GM-Fusion): In the target task, the model uses $X^{aux}$ as a query in a cross-attention mechanism. This "guide" dynamically adjusts the weights of different modalities based on the identified activity group, effectively filtering out noise and prioritizing the most relevant sensors.

Experimental Results and Discussion

The researchers evaluated MuMu on three benchmark datasets: MMAct, UTD-MHAD, and UCSD-MIT. These datasets cover a wide range of activities and sensor configurations.

Performance and Robustness

MuMu consistently outperformed state-of-the-art (SOTA) models. On the MMAct dataset, MuMu achieved a 76.28% F1-score in cross-subject testing, surpassing the previous best of 71.83%. Similar gains were observed on UTD-MHAD (97.60% accuracy).

A key strength of MuMu is its resilience to data corruption. By using the auxiliary task as a guide, the model can adaptively shift its focus when certain sensors become unreliable.

Guided Fusion Attention Weights under Noisy Conditions

As shown in the qualitative analysis, when visual and certain IMU data are noisy or missing, MuMu's guided mechanism identifies the activity group and prioritizes the reliable "Orientation" sensor. In contrast, standard self-attention-based fusion methods (like HAMLET) struggle to suppress the influence of noisy data streams. On noisy MMAct data, MuMu maintained a 73.78% F1-score, demonstrating significant stability compared to baseline models.

Conclusion

MuMu introduces a novel paradigm for multimodal fusion by incorporating high-level activity context via multitask learning. By "guiding" the fusion process, the model achieves superior accuracy and remarkable robustness against real-world sensor imperfections. Future work will explore unsupervised methods for discovering activity groups to enhance the system's applicability to "in-the-wild" scenarios.

Summary updated

Abstract

Multimodal sensors (visual, non-visual, and wearable) can provide complementary information to develop robust perception systems for recognizing activities accurately. However, it is challenging to extract robust multimodal representations due to the heterogeneous characteristics of data from multimodal sensors and disparate human activities, especially in the presence of noisy and misaligned sensor data. In this work, we propose a cooperative multitask learning-based guided multimodal fusion approach, MuMu, to extract robust multimodal representations for human activity recognition (HAR). MuMu employs an auxiliary task learning approach to extract features specific to each set of activities with shared characteristics (activity-group). MuMu then utilizes activity-group-specific features to direct our proposed Guided Multimodal Fusion Approach (GM-Fusion) for extracting complementary multimodal representations, designed as the target task. We evaluated MuMu by comparing its performance to state-of-the-art multimodal HAR approaches on three activity datasets. Our extensive experimental results suggest that MuMu outperforms all the evaluated approaches across all three datasets. Additionally, the ablation study suggests that MuMu significantly outperforms the baseline models (p
BibTeX
@inproceedings{islam2022mumu,
  author = {Md Mofijul Islam and Tariq Iqbal},
  title = {{MuMu: Cooperative Multitask Learning-Based Guided Multimodal Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=qtNggHfBuU5},
}
Figure