Introduction
Traditional video captioning models typically operate in a "one-shot" manner: they receive a video input and generate a single descriptive sentence. However, videos are semantically dense, and a single sentence often fails to capture the specific details a particular user might be interested in. For instance, in a video of a busy street, one user might want a description of the architecture, while another might be interested in the flow of traffic.
To address this, we introduce Video Interactive Captioning (ViCap), a human-in-the-loop framework. In this system, if an initial caption is unsatisfactory, a human provides a short natural language prompt (a "clue") to guide the agent toward a more specific and accurate description. This shifts the paradigm from static generation to a dynamic, interactive process.
The ViCap Framework
The core challenge of interactive captioning is effectively integrating three distinct sources of information: the raw video features, the initial (incorrect) caption, and the new human prompt. The ViCap agent employs several novel architectural choices to achieve this.
Convolutional Sequential Decoder
Unlike standard captioning models that rely on Recurrent Neural Networks (RNNs) like LSTMs or GRUs, ViCap utilizes a Convolutional Sequential Decoder. This decoder consists of five dilated convolutional layers. The hierarchical structure of CNNs allows the model to capture long-term dependencies more effectively than RNNs, which often suffer from "information dilution" or forgetting as sequences grow longer. As shown in the architecture, these layers allow the model to "see" the prompt and video features simultaneously across different time steps.
Enlarging the Semantic Gap
To prevent the model from simply repeating the errors of the initial caption, ViCap implements a "semantic gap" enlargement technique. The model calculates a "vertical representation" ($H_\perp$) that is mathematically orthogonal to the vector representation of the initial caption. By incorporating this orthogonal component during the decoding process, the model is forced to explore a different semantic space, ensuring the refined caption is distinct from the failed first attempt.
Weak Supervision with Recon-Net
Training interactive models is difficult due to the lack of large-scale interaction datasets. To mitigate this, the authors designed a Recurrent Reconstruction Network (Recon-Net). This component allows for "weak supervision" by forcing the decoder to learn representations capable of reconstructing the original video features. This ensures the model remains grounded in the visual content even when trained primarily on short human prompts.
Experimental Setup and Results
The framework was evaluated using an extended version of the MSRVTT-2016 dataset. Since standard datasets lack interaction data, the authors simulated prompts using the first two words of ground-truth target captions. The results demonstrate the superiority of the convolutional approach:
- Performance: The ViCap model achieved a CIDEr score of 46.04, significantly outperforming an improved GRU-based baseline (IGRU-D), which scored 38.50.
- Ablation Studies: Removing the orthogonal semantic gap component ($H_\perp$) led to a performance drop from 46.04 to 39.54, highlighting the importance of explicitly moving away from the initial caption's semantic space.
- Robustness: The model showed significant improvement even under weak supervision, proving that the Recon-Net effectively prevents overfitting to the short prompts while maintaining visual grounding.
Conclusion and Future Directions
ViCap represents a significant step toward more flexible and user-centric video captioning. By treating captioning as an interactive dialogue rather than a static task, the system can better serve diverse user needs. Future work will focus on handling more complex, natural language prompts beyond simple two-word "clues" and extending the framework to support multi-turn interactions until the user is fully satisfied.
Summary updated