Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living

Zdravko Marinov, David Schneider, Alina Roitberg, Rainer Stiefelhagen

This paper, presented at IROS 2022, tackles one of the most persistent "white whales" in robotics and computer vision: the Synthetic-to-Real gap. While training robots in simulation (like The Sims 4) is safe and cheap, models trained there often fail miserably—dropping over 60% in accuracy—when deployed in actual human homes.

TL;DR

Researchers from the Karlsruhe Institute of Technology (KIT) have developed a framework that bridges the synthetic-to-real gap by generating "novel action appearances." By taking synthetic data and mathematically forcing it to look like entirely new, unseen domains while preserving the core movement (pose, flow, etc.), they create a more robust "multimodal" training set that significantly improves activity recognition in real-world environments.

What Problem Does This Paper Address?

For assistive robots to be useful, they must recognize Activities of Daily Living (ADLs)—things like cooking, eating, or using a tablet. However, collecting thousands of hours of video in real homes is a privacy nightmare and incredibly expensive.

Simulations are the obvious solution, but they suffer from domain shift. A "cooking" action in a video game looks fundamentally different to a neural network than a "cooking" action in a grainy, poorly lit kitchen video. Standard models overfit to the specific textures and lighting of the simulation. The authors aim to move beyond simple data augmentation toward Domain Generalization (DG): training a model so that it learns the essence of an action, making it "immune" to changes in appearance.

Methods

The authors propose a multimodal generation framework. Instead of just looking at raw RGB pixels, they extract four "source modalities" from the synthetic video:

  1. Raw RGB: The original video frames.
  2. Heatmaps: 2D Gaussian maps centered on 17 human joint locations.
  3. Limbs: Skeleton-like lines connecting those joints.
  4. Optical Flow: The pattern of apparent motion between frames.

The Domain Generator (DG)

The core innovation is a Domain Generator network. It takes these source modalities and transforms them into "novel domains." To ensure these new domains are actually useful, the training uses a clever combination of losses:

  • Novelty & Diversity Loss: Using "Sinkhorn distance," the model is rewarded for making the new domains as different as possible from the source and from each other.
  • Classification Loss: A "frozen" action classifier ensures that even if the appearance changes wildly, the action (e.g., "walking") remains recognizable.
  • Reconstruction Loss: A cycle-consistency check to ensure the underlying structure of the image isn't lost in the transformation.

Finally, an Action Classifier (S3D architecture) is trained on a mix of the original synthetic data and these newly generated "hallucinated" domains, forcing it to ignore superficial appearance and focus on the multimodal motion.

Datasets

  • Training: Sims4Action, a dataset of 10 basic human actions recorded within the The Sims 4 video game.
  • Evaluation (Real World):
    • Toyota Smarthome: Real-world videos of 18 subjects in an apartment.
  • ETRI-Activity3D: Videos of 50 elderly subjects performing daily activities.

Key Insights and Ideas

  • Pose is the Bridge: The authors found that body-pose representations (heatmaps and limbs) are far more robust across domains than RGB. Using these alone provides a massive performance boost before any "generation" even happens.
  • Maximizing Distance: Most data augmentation tries to make data look "more real." This paper does the opposite: it tries to make data look as different as possible while keeping the label the same. This "adversarial" approach to domain generation forces the model to find the most invariant features.
  • Single-Source Multimodality: Crucially, the system doesn't require specialized sensors (like depth cameras or IMUs). All modalities (pose, flow, etc.) are inferred from a single standard RGB video stream.

Key Results

The framework sets a new state-of-the-art for the Synthetic→Real benchmark:

  • Accuracy Gains: On the Toyota Smarthome dataset, their best model achieved 27.73% balanced accuracy, compared to just 12.40% for a standard S3D model trained on raw synthetic RGB.
  • Outperforming Competitors: Their approach beat other established domain generalization methods (like TA3N and VideoDG) by significant margins (up to 5-10% in some metrics).
  • Diversification Works: Qualitative results (via t-SNE visualizations) showed that the generated novel modalities occupied a much wider, non-overlapping space in the latent distribution, proving the generator successfully "diversified" the training data.

Limitations and Future Work

  • Computational Overhead: Generating these novel domains and training on a "multiplied" dataset increases the computational cost and training time.
  • Pose Dependency: The system relies heavily on the quality of the initial pose estimation (AlphaPose). If the pose detector fails (e.g., due to heavy occlusion), the downstream modalities and generation will suffer.
  • Complexity Scaling: The authors noted that as the number of source modalities increases, the optimization becomes more difficult, potentially leading to a plateau in performance gains.

Future work could involve exploring how these generated domains might help in "zero-shot" scenarios where the model encounters an action it has never seen in any form during training.

Summary updated

Abstract

Domain shifts, such as appearance changes, are a key challenge in real-world applications of activity recognition models, which range from assistive robotics and smart homes to driver observation in intelligent vehicles. For example, while simulations are an excellent way of economical data collection, a Synthetic-to-Real domain shift leads to a > 60% drop in accuracy when recognizing activities of Daily Living (ADLs). We tackle this challenge and introduce an activity domain generation framework which creates novel ADL appearances (novel domains) from different existing activity modalities (source domains) inferred from video training data. Our framework computes human poses, heatmaps of body joints, and optical flow maps and uses them alongside the original RGB videos to learn the essence of source domains in order to generate completely new ADL domains. The model is optimized by maximizing the distance between the existing source appearances and the generated novel appearances while ensuring that the semantics of an activity is preserved through an additional classification loss. While source data multimodality is an important concept in this design, our setup does not rely on multi-sensor setups, (i.e., all source modalities are inferred from a single video only.) The newly created activity domains are then integrated in the training of the ADL classification networks, resulting in models far less susceptible to changes in data distributions. Extensive experiments on the Synthetic-to-Real benchmark Sims4Action demonstrate the potential of the domain generation paradigm for cross-domain ADL recognition, setting new state-of-the-art results. Our code is publicly available at https://github.com/Zrrr1997/syn2real_DG
BibTeX
@inproceedings{marinov2022multimodal,
  author = {Zdravko Marinov and David Schneider and Alina Roitberg and Rainer Stiefelhagen},
  title = {{Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981946},
}

Loading related content...

Figure