Deep Image-to-Video Adaptation and Fusion Networks for Action Recognition

Yang Liu, Zhaoyang Lu, Jing Li, Tao Yang, Chao Yao

Abstract

Existing deep learning methods for action recognition in videos require a large number of labeled videos for training, which is labor-intensive and time-consuming. For the same action, the knowledge learned from different media types, e.g., videos and images, may be related and complementary. However, due to the domain shifts and heterogeneous feature representations between videos and images, the performance of classifiers trained on images may be dramatically degraded when directly deployed to videos. In this paper, we propose a novel method, named Deep Image-to-Video Adaptation and Fusion Networks (DIVAFN), to enhance action recognition in videos by transferring knowledge from images using video keyframes as a bridge. The DIVAFN is a unified deep learning model, which integrates domain-invariant representations learning and cross-modal feature fusion into a unified optimization framework. Specifically, we design an efficient cross-modal similarities metric to reduce the modality shift among images, keyframes and videos. Then, we adopt an autoencoder architecture, whose hidden layer is constrained to be the semantic representations of the action class names. In this way, when the autoencoder is adopted to project the learned features from different domains to the same space, more compact, informative and discriminative representations can be obtained. Finally, the concatenation of the learned semantic feature representations from these three autoencoders are used to train the classifier for action recognition in videos. Comprehensive experiments on four real-world datasets show that our method outperforms some state-of-the-art domain adaptation and action recognition methods.

Introduction

Training deep learning models for video action recognition typically requires massive amounts of labeled video data—a process that is notoriously labor-intensive, time-consuming, and computationally expensive. In contrast, labeled images are much easier to acquire and are available in vast quantities.

However, a significant challenge arises when attempting to use image-based knowledge for video tasks: the domain shift. As shown in Figure 1, still images capture a single, representative "frozen" moment with high diversity in viewpoint and background. Motion videos, conversely, capture a temporal progression and often contain many redundant or uninformative frames. Furthermore, the feature representations are inherently heterogeneous, moving from 2D spatial data in images to 3D spatiotemporal data in videos.

Still images vs. motion videos for 'Horse riding' and 'Drinking' actions, illustrating the domain shift between media types.

The DIVAFN Framework

The proposed Deep Image-to-Video Adaptation and Fusion Network (DIVAFN) is a unified deep learning framework designed to bridge this gap. It integrates domain-invariant representation learning and cross-modal feature fusion to leverage image-based knowledge for improved video recognition.

Architecture Overview

The DIVAFN architecture consists of three parallel deep neural networks processing three distinct streams: static images, video keyframes, and the videos themselves. Keyframes serve as a critical bridge, acting as a natural middle ground between static images and temporal video data.

The DIVAFN framework architecture, integrating three parallel deep networks for images, keyframes, and videos with semantic autoencoders and cross-modal fusion.

Key Technical Innovations

  • Cross-Modal Similarity Metric: To reduce the "modality shift," the framework employs a metric that encourages features from the same action class to cluster together in a shared space, regardless of their source (image, keyframe, or video).
  • Semantic Autoencoders: This is the core of the adaptation process. The framework uses three autoencoders (Auto-F, Auto-H, and Auto-G) where the hidden layer is constrained to match the semantic representations of action class names. These representations are derived from Word2vec embeddings (e.g., Google News Skip-Gram models). This forces the model to project diverse visual features into a compact, informative space that reflects the underlying "meaning" of the action.
  • Feature Fusion: The final stage involves a cross-modal feature fusion autoencoder (Auto-E). The learned semantic features from the keyframe and video streams are concatenated and used to train an SVM classifier for the final action recognition task.

Experimental Evaluation

The study evaluated DIVAFN across several diverse image and video datasets, including standard benchmarks like UCF101 and HMDB51 for videos, and Stanford40 and BU101 for source images.

Key Results and Insights

  • State-of-the-Art Performance: DIVAFN consistently outperformed standard baselines and other popular domain adaptation methods (such as JDA, GFK, and JGSA) across all tested dataset combinations. On the BU101 → UCF101 task, DIVAFN achieved 88.4% accuracy.
  • Efficiency in Scarcity: A major strength of the framework is its performance when training videos are scarce. It achieved an impressive 86.1% accuracy on UCF101 using only 20% of the training videos, demonstrating its practical utility for real-world scenarios where video labels are rare.
  • Complementary Strengths: The results confirm that images provide "representative moments" while videos provide temporal context. Combining these leads to a more robust model than using either in isolation.
  • Robustness: The method proved effective regardless of whether the underlying video features were hand-crafted (IDT) or deeply learned (C3D), suggesting high generalizability.

Discussion and Future Work

While DIVAFN is powerful, there are areas for further exploration. The current architecture, involving three parallel networks and multiple autoencoders, is computationally heavier than single-stream approaches. Additionally, the performance is tied to the quality of the semantic embeddings; if the Word2vec model fails to capture the nuances of an action, the adaptation may be less effective. Future work could explore a fully end-to-end differentiable pipeline to further tighten the integration between feature extraction and classification.

Conclusion

DIVAFN offers a robust solution to the data scarcity problem in action recognition by effectively "borrowing" knowledge from labeled image datasets. By using keyframes as a bridge and enforcing semantic constraints through autoencoders, the framework significantly boosts recognition accuracy and demonstrates remarkable efficiency even with limited video data.

Summary updated

Abstract

Existing deep learning methods for action recognition in videos require a large number of labeled videos for training, which is labor-intensive and time-consuming. For the same action, the knowledge learned from different media types, e.g., videos and images, may be related and complementary. However, due to the domain shifts and heterogeneous feature representations between videos and images, the performance of classifiers trained on images may be dramatically degraded when directly deployed to videos. In this paper, we propose a novel method, named Deep Image-to-Video Adaptation and Fusion Networks (DIVAFN), to enhance action recognition in videos by transferring knowledge from images using video keyframes as a bridge. The DIVAFN is a unified deep learning model, which integrates domain-invariant representations learning and cross-modal feature fusion into a unified optimization framework. Specifically, we design an efficient cross-modal similarities metric to reduce the modality shift among images, keyframes and videos. Then, we adopt an autoencoder architecture, whose hidden layer is constrained to be the semantic representations of the action class names. In this way, when the autoencoder is adopted to project the learned features from different domains to the same space, more compact, informative and discriminative representations can be obtained. Finally, the concatenation of the learned semantic feature representations from these three autoencoders are used to train the classifier for action recognition in videos. Comprehensive experiments on four real-world datasets show that our method outperforms some state-of-the-art domain adaptation and action recognition methods.
BibTeX
@article{liu2020deep,
  author = {Yang Liu and Zhaoyang Lu and Jing Li and Tao Yang and Chao Yao},
  title = {{Deep Image-to-Video Adaptation and Fusion Networks for Action Recognition}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=wjhgUq36zc},
}
Figure