MoS$^2$: Mixture of Scale and Shift Experts for Text-Only Video Captioning

Heng Jia, Yunqiu Xu, Linchao Zhu, Guang Chen, Yufei Wang, Yi Yang

Abstract

Video captioning is a challenging task and typically requires video-text paired data for training. However, manually annotating coherent textual descriptions for videos is laborious and time-consuming. To address this problem, we propose to utilize solely text data to enhance video captioning models. Drawing inspiration from the exceptional text generation capabilities demonstrated by large language models (LLMs), we aim to leverage these models to generate high-quality and high-diversity video captions for the target domain. Specifically, we prompt GPT-4 with few-shot target-domain captions to generate a limited set of plausible video captions. Subsequently, we continue to prompt GPT-4 with the generated captions to acquire large-scale captions. To fully exploit the generated captions, we propose a Mixture of Scale and Shift experts (MoS$^2$) for efficient adaptation of pre-trained image captioning models for video captioning. MoS$^2$ estimates a probability distribution over a collection of experts by a lightweight routing network, determining the allocation of tokens to appropriate experts. This dynamic adjustment mechanism allows for specific responses to input features, thereby enhancing the model's ability to handle data variations. Our approach not only customizes model responses to input variations, effectively addressing the distribution shift between synthetic and actual captions but also significantly reduces the number of learnable parameters, allowing for more efficient adaptations. With only text data, we achieve superior performance and significantly narrow the performance gap between zero-shot and fine-tuned models. Our method boosts video captioning performance with the synthetic text data, thus substantially alleviating the dependence on paired and large-scale real data of the target domain. The code will be publicly available.
BibTeX
@inproceedings{jia2024mos,
  author = {Heng Jia and Yunqiu Xu and Linchao Zhu and Guang Chen and Yufei Wang and Yi Yang},
  title = {{MoS$^2$: Mixture of Scale and Shift Experts for Text-Only Video Captioning}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680686},
}

Loading related content...

Figure