ShiftFormer: Spatial-Temporal Shift Operation in Video Transformer

Beiying Yang, Guibo Zhu, Guojing Ge, Jinzhao Luo, Jinqiao Wang

Abstract

Transformers have achieved great success in various tasks, especially that introducing pure Transformers into video understanding shows powerful performance. However, video Transformer suffers from the problem of memory explosion: it is difficult to be deployed on hardware due to the intensive computation. To address this issue, we propose ST-shift (spatial-temporal) operation with zero computation and zero parameter. We are only shifting a small portion of the channels along the temporal and spatial dimensions. Based on this operation, we build an attention-free ShiftFormer, where ST-shift blocks substitute the attention layers in video Transformer. ShiftFormer is accurate and efficient: it can reduce 56.34% of memory usage and achieve 3.41× faster training. When both using random initialization, our model performs even better than Video Swin Transformer for video recognition on Something-Something v2.
BibTeX
@inproceedings{yang2023shiftformer,
  author = {Beiying Yang and Guibo Zhu and Guojing Ge and Jinzhao Luo and Jinqiao Wang},
  title = {{ShiftFormer: Spatial-Temporal Shift Operation in Video Transformer}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00325},
}

Loading related content...

Figure