Abstract
Multimodal sensors (visual, non-visual, and wearable) can provide complementary information to develop robust perception systems for recognizing activities accurately. However, it is challenging to extract robust multimodal representations due to the heterogeneous characteristics of data from multimodal sensors and disparate human activities, especially in the presence of noisy and misaligned sensor data. In this work, we propose a cooperative multitask learning-based guided multimodal fusion approach, MuMu, to extract robust multimodal representations for human activity recognition (HAR). MuMu employs an auxiliary task learning approach to extract features specific to each set of activities with shared characteristics (activity-group). MuMu then utilizes activity-group-specific features to direct our proposed Guided Multimodal Fusion Approach (GM-Fusion) for extracting complementary multimodal representations, designed as the target task. We evaluated MuMu by comparing its performance to state-of-the-art multimodal HAR approaches on three activity datasets. Our extensive experimental results suggest that MuMu outperforms all the evaluated approaches across all three datasets.
Introduction
Human Activity Recognition (HAR) is a cornerstone of modern assistive technologies, autonomous systems, and healthcare monitoring. By integrating diverse modalities—such as RGB video, depth maps, and inertial measurement unit (IMU) data—systems can achieve a more holistic understanding of human movement. However, two primary challenges persist: the heterogeneity of activities and the inherent "messiness" of real-world sensor data.
Different activities often rely on specific modalities; for instance, fine-grained hand movements are best captured by visual or wrist-worn sensors, while gross motor skills like running are more evident in leg-mounted IMUs. Furthermore, real-world deployments frequently encounter sensor noise, occlusions, and temporal misalignments. Standard fusion techniques often fail when a primary modality becomes unreliable.
Methodology: The MuMu Framework
MuMu addresses these challenges through a cooperative multitask learning architecture. The framework is designed to leverage high-level activity context to guide the fusion process.
The architecture comprises three core components:
- Unimodal Feature Encoders (UFE): These encoders (e.g., ResNet-50 for video, LSTMs for time-series) extract independent feature representations from each sensor modality.
- Auxiliary Task Learning (ATL): This module learns to classify broad "activity groups" (e.g., desk-based vs. complex movements). The resulting feature vector ($X^{aux}$) provides a high-level context of the action.
- Guided Multimodal Fusion (GM-Fusion): In the target task, the model uses $X^{aux}$ as a query in a cross-attention mechanism. This "guide" dynamically adjusts the weights of different modalities based on the identified activity group, effectively filtering out noise and prioritizing the most relevant sensors.
Experimental Results and Discussion
The researchers evaluated MuMu on three benchmark datasets: MMAct, UTD-MHAD, and UCSD-MIT. These datasets cover a wide range of activities and sensor configurations.
Performance and Robustness
MuMu consistently outperformed state-of-the-art (SOTA) models. On the MMAct dataset, MuMu achieved a 76.28% F1-score in cross-subject testing, surpassing the previous best of 71.83%. Similar gains were observed on UTD-MHAD (97.60% accuracy).
A key strength of MuMu is its resilience to data corruption. By using the auxiliary task as a guide, the model can adaptively shift its focus when certain sensors become unreliable.
As shown in the qualitative analysis, when visual and certain IMU data are noisy or missing, MuMu's guided mechanism identifies the activity group and prioritizes the reliable "Orientation" sensor. In contrast, standard self-attention-based fusion methods (like HAMLET) struggle to suppress the influence of noisy data streams. On noisy MMAct data, MuMu maintained a 73.78% F1-score, demonstrating significant stability compared to baseline models.
Conclusion
MuMu introduces a novel paradigm for multimodal fusion by incorporating high-level activity context via multitask learning. By "guiding" the fusion process, the model achieves superior accuracy and remarkable robustness against real-world sensor imperfections. Future work will explore unsupervised methods for discovering activity groups to enhance the system's applicability to "in-the-wild" scenarios.
Summary updated