Deep Image-to-Video Adaptation and Fusion Networks for Action Recognition

Yang Liu, Zhaoyang Lu, Jing Li, Tao Yang, Chao Yao

TL;DR

The Deep Image-to-Video Adaptation and Fusion Network (DIVAFN) tackles the high cost of labeling video data by "borrowing" knowledge from more abundant labeled image datasets. By using video keyframes as a bridge and enforcing semantic constraints through autoencoders, the framework significantly boosts action recognition accuracy, especially when training videos are scarce.

What Problem Does This Paper Address?

Training deep learning models for video action recognition typically requires massive amounts of labeled video data—a process that is notoriously labor-intensive, time-consuming, and computationally expensive. In contrast, labeled images are much easier to acquire.

However, you can't just train a model on images and expect it to work perfectly on videos. There is a significant domain shift: images capture a single, representative "frozen" moment with high diversity in viewpoint and background, while videos capture a temporal progression and often contain many redundant or uninformative frames. Furthermore, the feature representations are heterogeneous (2D spatial vs. 3D spatiotemporal). The authors aim to bridge this gap, allowing models to leverage image-based knowledge to improve video recognition.

Methods

The authors propose DIVAFN, a unified deep learning framework that integrates domain-invariant representation learning and cross-modal feature fusion. Its architecture consists of three parallel deep neural networks (one for images, one for video keyframes, and one for the videos themselves).

Key technical innovations include:

  • Keyframes as a Bridge: Since a video is a sequence of frames, keyframes serve as a natural middle ground between static images and temporal video data.
  • Cross-Modal Similarity Metric: To reduce the "modality shift," the authors designed a metric that encourages features from the same action class to cluster together in a shared space, regardless of whether they originated from an image, a keyframe, or a video.
  • Semantic Autoencoders: This is the "secret sauce." The framework uses autoencoders where the hidden layer is constrained to match the semantic representations of action class names (using word2vec embeddings or human-labeled attributes). This forces the model to project diverse visual features into a compact, informative, and discriminative space that reflects the actual "meaning" of the action.
  • Feature Fusion: The final classifier (an SVM) is trained on the concatenation of the learned semantic representations from the keyframe, video, and a specifically designed fusion autoencoder.

Datasets

The study evaluates the method across several diverse image and video datasets:

  • Target Video Datasets: UCF101 and HMDB51 (standard benchmarks for action recognition).
  • Source Image Datasets:
    • Stanford40: 40 human action categories.
  • BU101: A large-scale web image dataset with a one-to-one correspondence to UCF101 classes.
  • ASD (Actions from Still Datasets): A combination of Action DB, PPMI, Willow-Actions, and Still DB.
  • EAD (Extensive Action Dataset): A custom mix of Stanford40 and the Human Interaction Image (HII) dataset.

Key Insights and Ideas

  • Complementary Strengths: The authors observe that images focus on "representative moments" (a single glance is often enough to identify the action), while videos provide the full temporal context. Combining these leads to a more robust model than using either in isolation.
  • Semantic Guidance: By forcing the latent space of the autoencoders to align with word embeddings (like word2vec), the model learns features that are less tied to specific visual noise and more tied to the underlying concept of the action.
  • Efficiency in Scarcity: The most striking insight is how well this works with limited data. The framework allows for high performance even when only 10% or 20% of the available video training data is used, making it highly practical for real-world applications where video labels are rare.

Key Results

  • State-of-the-Art Performance: DIVAFN consistently outperformed standard baselines and other popular domain adaptation methods (like JDA, GFK, and JGSA) across all four tested dataset combinations.
  • Large-Scale Success: On the BU101 $\rightarrow$ UCF101 task, DIVAFN achieved 88.4% accuracy using 100% of the training videos, and a impressive 86.1% using only 20% of the training videos.
  • Robustness: The method proved effective regardless of whether the underlying video features were hand-crafted (IDT) or deeply learned (C3D), suggesting the framework is highly generalizable.

Limitations and Future Work

While DIVAFN is powerful, the paper notes or implies a few areas for further exploration:

  • Computational Complexity: Training three parallel deep networks and multiple autoencoders is computationally heavier than a single-stream approach.
  • Semantic Dependency: The performance is somewhat dependent on the quality of the semantic embeddings (word2vec or attributes). If the embeddings don't capture the nuances of the actions well, the "bridge" might weaken.
  • End-to-End Potential: While the feature extraction and domain adaptation are integrated, the final classification still relies on a separate SVM. Future iterations might look into a fully end-to-end differentiable pipeline for even tighter integration.

Summary updated

Abstract

Existing deep learning methods for action recognition in videos require a large number of labeled videos for training, which is labor-intensive and time-consuming. For the same action, the knowledge learned from different media types, e.g., videos and images, may be related and complementary. However, due to the domain shifts and heterogeneous feature representations between videos and images, the performance of classifiers trained on images may be dramatically degraded when directly deployed to videos. In this paper, we propose a novel method, named Deep Image-to-Video Adaptation and Fusion Networks (DIVAFN), to enhance action recognition in videos by transferring knowledge from images using video keyframes as a bridge. The DIVAFN is a unified deep learning model, which integrates domain-invariant representations learning and cross-modal feature fusion into a unified optimization framework. Specifically, we design an efficient cross-modal similarities metric to reduce the modality shift among images, keyframes and videos. Then, we adopt an autoencoder architecture, whose hidden layer is constrained to be the semantic representations of the action class names. In this way, when the autoencoder is adopted to project the learned features from different domains to the same space, more compact, informative and discriminative representations can be obtained. Finally, the concatenation of the learned semantic feature representations from these three autoencoders are used to train the classifier for action recognition in videos. Comprehensive experiments on four real-world datasets show that our method outperforms some state-of-the-art domain adaptation and action recognition methods.
BibTeX
@article{liu2020deep,
  author = {Yang Liu and Zhaoyang Lu and Jing Li and Tao Yang and Chao Yao},
  title = {{Deep Image-to-Video Adaptation and Fusion Networks for Action Recognition}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=wjhgUq36zc},
}

Loading related content...

Figure