Video Interactive Captioning with Human Prompts

Aming Wu, Yahong Han, Yi Yang

Introduction

Traditional video captioning models typically operate in a "one-shot" manner: they receive a video input and generate a single descriptive sentence. However, videos are semantically dense, and a single sentence often fails to capture the specific details a particular user might be interested in. For instance, in a video of a busy street, one user might want a description of the architecture, while another might be interested in the flow of traffic.

Overview of the interactive captioning process between a human and the ViCap agent.

To address this, we introduce Video Interactive Captioning (ViCap), a human-in-the-loop framework. In this system, if an initial caption is unsatisfactory, a human provides a short natural language prompt (a "clue") to guide the agent toward a more specific and accurate description. This shifts the paradigm from static generation to a dynamic, interactive process.

The ViCap Framework

The core challenge of interactive captioning is effectively integrating three distinct sources of information: the raw video features, the initial (incorrect) caption, and the new human prompt. The ViCap agent employs several novel architectural choices to achieve this.

Convolutional Sequential Decoder

Unlike standard captioning models that rely on Recurrent Neural Networks (RNNs) like LSTMs or GRUs, ViCap utilizes a Convolutional Sequential Decoder. This decoder consists of five dilated convolutional layers. The hierarchical structure of CNNs allows the model to capture long-term dependencies more effectively than RNNs, which often suffer from "information dilution" or forgetting as sequences grow longer. As shown in the architecture, these layers allow the model to "see" the prompt and video features simultaneously across different time steps.

The ViCap architecture featuring a convolutional sequential decoder with dilated layers.

Enlarging the Semantic Gap

To prevent the model from simply repeating the errors of the initial caption, ViCap implements a "semantic gap" enlargement technique. The model calculates a "vertical representation" ($H_\perp$) that is mathematically orthogonal to the vector representation of the initial caption. By incorporating this orthogonal component during the decoding process, the model is forced to explore a different semantic space, ensuring the refined caption is distinct from the failed first attempt.

Weak Supervision with Recon-Net

Training interactive models is difficult due to the lack of large-scale interaction datasets. To mitigate this, the authors designed a Recurrent Reconstruction Network (Recon-Net). This component allows for "weak supervision" by forcing the decoder to learn representations capable of reconstructing the original video features. This ensures the model remains grounded in the visual content even when trained primarily on short human prompts.

Experimental Setup and Results

The framework was evaluated using an extended version of the MSRVTT-2016 dataset. Since standard datasets lack interaction data, the authors simulated prompts using the first two words of ground-truth target captions. The results demonstrate the superiority of the convolutional approach:

  • Performance: The ViCap model achieved a CIDEr score of 46.04, significantly outperforming an improved GRU-based baseline (IGRU-D), which scored 38.50.
  • Ablation Studies: Removing the orthogonal semantic gap component ($H_\perp$) led to a performance drop from 46.04 to 39.54, highlighting the importance of explicitly moving away from the initial caption's semantic space.
  • Robustness: The model showed significant improvement even under weak supervision, proving that the Recon-Net effectively prevents overfitting to the short prompts while maintaining visual grounding.

Conclusion and Future Directions

ViCap represents a significant step toward more flexible and user-centric video captioning. By treating captioning as an interactive dialogue rather than a static task, the system can better serve diverse user needs. Future work will focus on handling more complex, natural language prompts beyond simple two-word "clues" and extending the framework to support multi-turn interactions until the user is fully satisfied.

Summary updated

Abstract

Video captioning aims at generating a proper sentence to describe the video content. As a video often includes rich visual content and semantic details, different people may be interested in different views. Thus the generated sentence always fails to meet the ad hoc expectations. In this paper, we make a new attempt that, we launch a round of interaction between a human and a captioning agent. After generating an initial caption, the agent asks for a short prompt from the human as a clue of his expectation. Then, based on the prompt, the agent could generate a more accurate caption. We name this process a new task of video interactive captioning (ViCap). Taking a video and an initial caption as input, we devise the ViCap agent which consists of a video encoder, an initial caption encoder, and a refined caption generator. We show that the ViCap can be trained via a full supervision (with ground-truth) way or a weak supervision (with only prompts) way. For the evaluation of ViCap, we first extend the MSRVTT with interaction ground-truth. Experimental results not only show the prompts can help generate more accurate captions, but also demonstrate the good performance of the proposed method.
BibTeX
@inproceedings{wu2019video,
  author = {Aming Wu and Yahong Han and Yi Yang},
  title = {{Video Interactive Captioning with Human Prompts}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/135},
}
Figure