Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living

Zdravko Marinov, David Schneider, Alina Roitberg, Rainer Stiefelhagen

Abstract

Domain shifts, such as appearance changes, are a key challenge in real-world applications of activity recognition models, which range from assistive robotics and smart homes to driver observation in intelligent vehicles. For example, while simulations are an excellent way of economical data collection, a Synthetic-to-Real domain shift leads to a > 60% drop in accuracy when recognizing activities of Daily Living (ADLs). We tackle this challenge and introduce an activity domain generation framework which creates novel ADL appearances (novel domains) from different existing activity modalities (source domains) inferred from video training data. Our framework computes human poses, heatmaps of body joints, and optical flow maps and uses them alongside the original RGB videos to learn the essence of source domains in order to generate completely new ADL domains. The model is optimized by maximizing the distance between the existing source appearances and the generated novel appearances while ensuring that the semantics of an activity is preserved through an additional classification loss. The newly created activity domains are then integrated in the training of the ADL classification networks, resulting in models far less susceptible to changes in data distributions. Extensive experiments on the Synthetic-to-Real benchmark Sims4Action demonstrate the potential of the domain generation paradigm for cross-domain ADL recognition, setting new state-of-the-art results.

Introduction

In the field of robotics and computer vision, the "Synthetic-to-Real gap" remains a significant barrier to deploying robust activity recognition systems. While training models in simulated environments like The Sims 4 offers a safe and cost-effective alternative to real-world data collection, these models often suffer from a dramatic decline in performance—sometimes exceeding a 60% drop in accuracy—when tested in actual human homes. This failure is primarily due to domain shift: the textures, lighting, and backgrounds of a simulation differ fundamentally from the messy, varied environments of real life.

To address this, researchers from the Karlsruhe Institute of Technology (KIT) have proposed a framework that moves beyond simple data augmentation toward true Domain Generalization (DG). By generating "novel action appearances" that preserve the underlying semantics of an activity, they enable models to learn the invariant essence of human motion.

Methodology

The proposed framework utilizes a multimodal approach to bridge the domain gap. Rather than relying solely on raw RGB pixels, which are highly sensitive to environment-specific textures, the system extracts four distinct "source modalities" from synthetic video: raw RGB, 2D Gaussian heatmaps of body joints, skeleton-like limb connections, and optical flow maps representing motion patterns.

Overview of the proposed framework, showing modality extraction and the generation of novel action domains.

The Domain Generator (DG) Network

The core of the system is a Domain Generator network designed to transform these source modalities into entirely new, unseen domains. The training process is governed by three primary objectives:

  1. Novelty and Diversity: Using Sinkhorn distance, the model is incentivized to make generated domains as distinct as possible from the source data and from each other.
  2. Semantic Preservation: A frozen action classifier ensures that despite radical changes in appearance, the core activity (e.g., "cooking" or "walking") remains identifiable.
  3. Structural Consistency: A reconstruction loss maintains the underlying physical structure of the human subject during the transformation.

Crucially, this system achieves multimodality without requiring specialized hardware like depth sensors or IMUs; all modalities are inferred from a single standard RGB video stream.

Experimental Results

The framework was evaluated on the Sims4Action synthetic dataset and tested against real-world benchmarks, including Toyota Smarthome and ETRI-Activity3D.

Real-world activity recognition results on the Toyota Smarthome dataset, demonstrating the model's ability to predict daily activities.

The results demonstrate a significant leap in performance. On the Toyota Smarthome dataset, the proposed framework achieved a balanced accuracy of 27.73%, more than doubling the 12.40% achieved by a standard S3D model trained only on synthetic RGB data. Furthermore, the approach outperformed established domain generalization methods like TA3N and VideoDG by margins of 5-10%.

Conclusion

By forcing models to train on a diverse set of "hallucinated" domains that emphasize motion and pose over superficial textures, this research provides a scalable path toward robust activity recognition in robotics. While challenges remain regarding computational overhead and the dependency on initial pose estimation quality, the success of this multimodal domain generation paradigm sets a new state-of-the-art for synthetic-to-real transfer in activities of daily living.

Summary updated

Abstract

Domain shifts, such as appearance changes, are a key challenge in real-world applications of activity recognition models, which range from assistive robotics and smart homes to driver observation in intelligent vehicles. For example, while simulations are an excellent way of economical data collection, a Synthetic-to-Real domain shift leads to a > 60% drop in accuracy when recognizing activities of Daily Living (ADLs). We tackle this challenge and introduce an activity domain generation framework which creates novel ADL appearances (novel domains) from different existing activity modalities (source domains) inferred from video training data. Our framework computes human poses, heatmaps of body joints, and optical flow maps and uses them alongside the original RGB videos to learn the essence of source domains in order to generate completely new ADL domains. The model is optimized by maximizing the distance between the existing source appearances and the generated novel appearances while ensuring that the semantics of an activity is preserved through an additional classification loss. While source data multimodality is an important concept in this design, our setup does not rely on multi-sensor setups, (i.e., all source modalities are inferred from a single video only.) The newly created activity domains are then integrated in the training of the ADL classification networks, resulting in models far less susceptible to changes in data distributions. Extensive experiments on the Synthetic-to-Real benchmark Sims4Action demonstrate the potential of the domain generation paradigm for cross-domain ADL recognition, setting new state-of-the-art results. Our code is publicly available at https://github.com/Zrrr1997/syn2real_DG
BibTeX
@inproceedings{marinov2022multimodal,
  author = {Zdravko Marinov and David Schneider and Alina Roitberg and Rainer Stiefelhagen},
  title = {{Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981946},
}
Figure