Real-Time Action Recognition With Deeply Transferred Motion Vector CNNs
Lists
Only you see this
Abstract
The two-stream CNNs prove very successful for video-based action recognition. However, the classical two-stream CNNs are time costly, mainly due to the bottleneck of calculating optical flows (OFs). In this paper, we propose a two-stream-based real-time action recognition approach by using motion vector (MV) to replace OF. MVs are encoded in video stream and can be extracted directly without extra calculation. However, directly training CNN with MVs degrades accuracy severely due to the noise and the lack of fine details in MVs. In order to relieve this problem, we propose four training strategies which leverage the knowledge learned from OF CNN to enhance the accuracy of MV CNN. Our insight is that MV and OF share inherent similar structures which allow us to transfer knowledge from one domain to another. To fully utilize the knowledge learned in OF domain, we develop deeply transferred MV CNN. Experimental results on various datasets show the effectiveness of our training strategies. Our approach is significantly faster than OF based approaches and achieves processing speed of 390.7 frames per second, surpassing real-time requirement. We release our model and code to facilitate further research.
BibTeX
@article{zhang2018real,
author = {Bowen Zhang and Limin Wang and Zhe Wang and Yu Qiao and Hanli Wang},
title = {{Real-Time Action Recognition With Deeply Transferred Motion Vector CNNs}},
journal = {IEEE Trans. Image Process.},
year = {2018},
doi = {10.1109/tip.2018.2791180},
}Loading related content...