Dual-Path Adaptation from Image to Video Transformers

Jungin Park, Jiyoung Lee, Kwanghoon SohnCVPR 2023 · OpenReview

Abstract

Adapting massive image foundation models, such as Vision Transformers (ViT), for video understanding typically requires significant computational resources or compromises on temporal modeling. This paper discusses DUALPATH, a parameter-efficient transfer learning (PETL) framework that transforms frozen 2D image models into high-performance video classifiers. By bifurcating the adaptation into specialized spatial and temporal paths and utilizing a novel grid-like frameset, DUALPATH achieves state-of-the-art results on major benchmarks with a fraction of the trainable parameters and computational overhead.

Introduction

Vision Transformers (ViT) have established a new paradigm in image recognition, yet their application to video remains a challenge. Video understanding necessitates capturing both spatial context (object appearance) and temporal context (motion and sequence). The conventional approach of full fine-tuning on video datasets is computationally prohibitive and risks losing the robust features learned during image pre-training.

Recent Parameter-Efficient Transfer Learning (PETL) methods attempt to mitigate this by adding small, trainable "adapters" to frozen models. However, many existing PETL strategies either fail to capture complex temporal dynamics or introduce heavy 3D components that increase inference costs. DUALPATH addresses this by "tricking" 2D models into performing 3D reasoning through a dual-stream architecture and a clever data representation strategy.

Methodology

DUALPATH splits the adaptation process into two distinct, specialized streams that operate alongside the frozen transformer backbone:

1. The Spatial Path

This path is dedicated to maintaining and refining the model's understanding of appearance. It processes sparsely sampled frames from the video. To adapt the frozen backbone, the authors employ parallel adapters—lightweight bottleneck layers that run in parallel with the original transformer blocks. This structure allows the model to preserve its pre-trained image knowledge while learning task-specific spatial features.

2. The Temporal Path

The core innovation of DUALPATH is its approach to motion. Instead of using 3D convolutions, the authors introduce a grid-like frameset. Consecutive video frames are downscaled and tiled into a single large image (a grid).

By feeding this grid into the transformer, the model's existing 2D self-attention mechanism naturally learns relationships between the tiled frames. It treats temporal correlations across time as if they were spatial correlations across a single image. This path utilizes serial adapters (placed after the attention and MLP layers) to more aggressively transform the representations for temporal reasoning.

By freezing the massive backbone and only training these lightweight adapters and a final classifier, the model retains its powerful image-recognition foundation while gaining sophisticated video understanding capabilities.

Experimental Results

The researchers evaluated DUALPATH across four diverse action recognition benchmarks: Kinetics-400 (general actions), Something-Something-v2 (temporal-heavy actions), HMDB51 (domain generalization), and Diving-48 (fine-grained motion).

Efficiency and Performance

On the Kinetics-400 dataset, DUALPATH (using a ViT-B backbone) achieved a 85.4% Top-1 accuracy. Remarkably, this was achieved with only 10M trainable parameters and 710 GFLOPs. In comparison, the popular ST-Adapter achieves 82.7% accuracy while requiring over 1800 GFLOPs, making DUALPATH significantly more efficient.

Temporal Reasoning

On the Something-Something-v2 (SSv2) dataset, where the order of frames is critical for correct classification (e.g., distinguishing "opening a box" from "closing a box"), DUALPATH outperformed many fully fine-tuned models. It proved to be approximately 13x more efficient in terms of inference computation than competing PETL methods like EVL.

Robustness in Low-Data Regimes

On the HMDB51 dataset, DUALPATH improved performance by 9.2% over previous state-of-the-art efficient methods. This suggests that the dual-path approach is particularly effective when training data is scarce, as it leverages the frozen backbone's priors more effectively than methods that require more extensive training.

Discussion and Limitations

While DUALPATH offers significant gains in efficiency, it involves a trade-off regarding spatial resolution. The grid-like frameset requires downscaling individual frames to fit them into a single tiled image, which may result in the loss of fine-grained spatial details necessary for identifying very small objects.

Future research may explore extending this dual-path architecture to more complex multimodal tasks, such as video-language modeling or video object segmentation. Additionally, finding ways to incorporate minimal 3D spatial priors without sacrificing the 2D efficiency of the current model remains a promising avenue for development.

Conclusion

DUALPATH demonstrates that high-performance video understanding does not require massive 3D architectures or expensive fine-tuning. By cleverly repurposing 2D self-attention for temporal reasoning through frame tiling and using specialized adapter paths, DUALPATH sets a new standard for parameter-efficient video adaptation.

Summary updated

Abstract

In this paper, we efficiently transfer the surpassing representation power of the vision foundation models, such as ViT and Swin, for video understanding with only a few trainable parameters. Previous adaptation methods have simultaneously considered spatial and temporal modeling with a unified learnable module but still suffered from fully leveraging the representative capabilities of image transformers. We argue that the popular dual-path (two-stream) architecture in video models can mitigate this problem. We propose a novel DualPath adaptation separated into spatial and temporal adaptation paths, where a lightweight bottleneck adapter is employed in each transformer block. Especially for temporal dynamic modeling, we incorporate consecutive frames into a grid-like frameset to precisely imitate vision transformers' capability that extrapolates relationships between tokens. In addition, we extensively investigate the multiple baselines from a unified perspective in video understanding and compare them with DualPath. Experimental results on four action recognition benchmarks prove that pretrained image transformers with DualPath can be effectively generalized beyond the data domain.
BibTeX
@inproceedings{park2023dual,
  author = {Jungin Park and Jiyoung Lee and Kwanghoon Sohn},
  title = {{Dual-Path Adaptation from Image to Video Transformers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=JG4yYsXL-Qy},
}
Figure