Abstract
Domain shifts, such as appearance changes, are a key challenge in real-world applications of activity recognition models, which range from assistive robotics and smart homes to driver observation in intelligent vehicles. For example, while simulations are an excellent way of economical data collection, a Synthetic-to-Real domain shift leads to a > 60% drop in accuracy when recognizing activities of Daily Living (ADLs). We tackle this challenge and introduce an activity domain generation framework which creates novel ADL appearances (novel domains) from different existing activity modalities (source domains) inferred from video training data. Our framework computes human poses, heatmaps of body joints, and optical flow maps and uses them alongside the original RGB videos to learn the essence of source domains in order to generate completely new ADL domains. The model is optimized by maximizing the distance between the existing source appearances and the generated novel appearances while ensuring that the semantics of an activity is preserved through an additional classification loss. The newly created activity domains are then integrated in the training of the ADL classification networks, resulting in models far less susceptible to changes in data distributions. Extensive experiments on the Synthetic-to-Real benchmark Sims4Action demonstrate the potential of the domain generation paradigm for cross-domain ADL recognition, setting new state-of-the-art results.
Introduction
In the field of robotics and computer vision, the "Synthetic-to-Real gap" remains a significant barrier to deploying robust activity recognition systems. While training models in simulated environments like The Sims 4 offers a safe and cost-effective alternative to real-world data collection, these models often suffer from a dramatic decline in performance—sometimes exceeding a 60% drop in accuracy—when tested in actual human homes. This failure is primarily due to domain shift: the textures, lighting, and backgrounds of a simulation differ fundamentally from the messy, varied environments of real life.
To address this, researchers from the Karlsruhe Institute of Technology (KIT) have proposed a framework that moves beyond simple data augmentation toward true Domain Generalization (DG). By generating "novel action appearances" that preserve the underlying semantics of an activity, they enable models to learn the invariant essence of human motion.
Methodology
The proposed framework utilizes a multimodal approach to bridge the domain gap. Rather than relying solely on raw RGB pixels, which are highly sensitive to environment-specific textures, the system extracts four distinct "source modalities" from synthetic video: raw RGB, 2D Gaussian heatmaps of body joints, skeleton-like limb connections, and optical flow maps representing motion patterns.
The Domain Generator (DG) Network
The core of the system is a Domain Generator network designed to transform these source modalities into entirely new, unseen domains. The training process is governed by three primary objectives:
- Novelty and Diversity: Using Sinkhorn distance, the model is incentivized to make generated domains as distinct as possible from the source data and from each other.
- Semantic Preservation: A frozen action classifier ensures that despite radical changes in appearance, the core activity (e.g., "cooking" or "walking") remains identifiable.
- Structural Consistency: A reconstruction loss maintains the underlying physical structure of the human subject during the transformation.
Crucially, this system achieves multimodality without requiring specialized hardware like depth sensors or IMUs; all modalities are inferred from a single standard RGB video stream.
Experimental Results
The framework was evaluated on the Sims4Action synthetic dataset and tested against real-world benchmarks, including Toyota Smarthome and ETRI-Activity3D.
The results demonstrate a significant leap in performance. On the Toyota Smarthome dataset, the proposed framework achieved a balanced accuracy of 27.73%, more than doubling the 12.40% achieved by a standard S3D model trained only on synthetic RGB data. Furthermore, the approach outperformed established domain generalization methods like TA3N and VideoDG by margins of 5-10%.
Conclusion
By forcing models to train on a diverse set of "hallucinated" domains that emphasize motion and pose over superficial textures, this research provides a scalable path toward robust activity recognition in robotics. While challenges remain regarding computational overhead and the dependency on initial pose estimation quality, the success of this multimodal domain generation paradigm sets a new state-of-the-art for synthetic-to-real transfer in activities of daily living.
Summary updated