MoExDA: Domain Adaptation for Edge-based Action Recognition

Takuya Sugimoto, Ning Ding, Toru Tamaki

Introduction

Action recognition models often suffer from a phenomenon known as static bias, where they rely on scene appearance and static backgrounds rather than the actual temporal dynamics of an action. For example, a model might correctly classify a "golf swing" simply because it detects green grass, failing to generalize when the same action occurs in an unusual context.

To address this, we introduce MoExDA (Moment Exchange Domain Adaptation), a lightweight two-stream architecture that combines standard RGB frames with edge-detected frames. By utilizing a novel moment exchange module, MoExDA bridges the gap between texture-rich RGB data and structure-focused edge data, resulting in a model that is significantly more robust to background changes and focused on human dynamics.

Methodology

The MoExDA framework is built upon a two-stream Vision Transformer (ViT) architecture.

  1. Two-Stream Input: The first stream processes standard RGB frames. The second stream processes edge frames generated on-the-fly using a lightweight Sobel filter. These edge frames naturally strip away texture and color, leaving only the structural outlines of moving actors.
  2. Input Normalization: To account for the distinct pixel distributions of edge frames compared to RGB, specific mean and variance statistics from the Kinetics400 dataset are used for normalization.
  3. The MoExDA Module: This core component sits between the layers of the two streams. It calculates the moments (mean and standard deviation) of features in one stream and applies them to the other. Specifically, the "edge-to-RGB" configuration aligns edge features to the RGB distribution, allowing the model to "translate" structural information into a format compatible with RGB-trained layers.
MoExDA Architecture: A two-stream Vision Transformer (ViT) incorporating the Moment Exchange Domain Adaptation module between layers.

Normalization Strategies

The authors experimented with different normalization techniques, finding that Positional Normalization (PONO)—which normalizes locally at each image position—outperformed global Instance Normalization (IN). Because edge information is highly localized, PONO more effectively matches local edge moments with local RGB moments.

Experiments and Results

The model was evaluated using Kinetics50 for training and the Mimetics dataset for testing robustness against static bias. Mimetics contains actions performed in unusual contexts (e.g., juggling in a living room), which reveals if a model is over-relying on background cues. Performance was measured using Top-1 accuracy, BOR (Background-Only Ratio), and HOR (Human-Only Ratio).

Quantitative results comparing various MoExDA configurations (using IN and PONO) against baseline ViT models.

Key Findings

  • Reduced Static Bias: MoExDA significantly lowered the BOR compared to baseline ViT models. While the baseline RGB model had a BOR of ~83.6%, MoExDA configurations frequently dropped this below 70%, indicating a reduced reliance on background features.
  • Improved Robustness: On the Mimetics dataset, the proposed method showed a 2% to 5% improvement in top-1 accuracy over baseline models using only edge input.
  • Efficiency: By using simple edge detection and a lightweight exchange module, MoExDA avoids the heavy computational overhead associated with complex optical flow techniques.

Conclusion

MoExDA provides an efficient and effective solution for mitigating static bias in action recognition. While there is a slight trade-off between raw accuracy on biased datasets and generalization, the reduction in BOR demonstrates that the model focuses more accurately on human motion. Future work will explore estimating moments during training to enable edge-only inference, further optimizing the model for deployment on edge devices.

Summary updated

BibTeX
@inproceedings{sugimoto2025moexda,
  author = {Takuya Sugimoto and Ning Ding and Toru Tamaki},
  title = {{MoExDA: Domain Adaptation for Edge-based Action Recognition}},
  booktitle = {MVA},
  year = {2025},
  doi = {10.23919/mva65244.2025.11175123},
}
Figure