ContCap: A comprehensive framework for continual image captioning

Giang Nguyen, Tae Joon Jun, Trung Tran, Daeyoung KimarXiv 2019 · OpenReview

TL;DR

This paper introduces ContCap, a novel framework that tackles the challenge of continual image captioning. It addresses catastrophic forgetting in image captioning models by integrating continual learning techniques, enabling them to learn new tasks without degrading performance on previously learned ones.

What Problem Does This Paper Address?

Image captioning systems have become remarkably adept at generating descriptive text for images. However, when these models are trained to learn new tasks sequentially (e.g., learning to caption cats, then dogs, then birds), they often suffer from "catastrophic forgetting." This means that as the model learns new information, it completely loses the ability to perform well on tasks it was previously trained on. This is a significant hurdle for real-world applications where models need to adapt and learn over time without access to all past training data. The paper aims to bridge this gap by developing a framework that allows image captioning models to learn incrementally.

Methods

The core of ContCap is its integration of continual learning strategies into an encoder-decoder image captioning architecture. The authors propose several techniques to combat catastrophic forgetting:

  • Pseudo-labeling: This method uses the predictions of the previous model on new data to generate "pseudo-labels." These pseudo-labels then guide the training of the new model, helping it retain knowledge from older tasks.
  • Partial Freezing: This involves freezing either the encoder (feature extractor) or the decoder (text generator) while training the other part. This helps preserve learned features or generation capabilities from previous tasks.
  • Feature Distillation: Here, the knowledge from a previously trained model (the "teacher") is transferred to the new model (the "student") by minimizing the difference in their intermediate feature representations. This encourages the new model to produce similar outputs to the old one.

These techniques are designed to be modular and can be integrated into various encoder-decoder architectures, making ContCap a scalable solution.

Datasets

The experiments were conducted on a newly created dataset called Split MS-COCO. This dataset is derived from the standard MS-COCO 2014 benchmark. To create Split MS-COCO, the authors:

  1. Split images into distinct classes.
  2. Resized images to a uniform (224x224) resolution.
  3. Discarded images containing more than one class in their annotations to create "clear" images (images primarily associated with a single class).

This dataset was then further divided to simulate class-incremental learning scenarios, where tasks are introduced sequentially with new, previously unseen classes.

Key Insights and Ideas

  • Catastrophic forgetting is a real problem in image captioning: The paper provides clear empirical evidence of this phenomenon, showing significant performance degradation on old tasks after learning new ones.
  • Pseudo-labeling shows promise for retaining old knowledge: This technique appears to be particularly effective in preventing performance drops on previously learned tasks, even when dealing with multiple new classes.
  • Freezing strategies offer a trade-off: While freezing parts of the model can help, it doesn't always lead to the best overall performance, suggesting a delicate balance is needed.
  • Sequential addition of classes is significantly harder: Learning classes one by one leads to a much more severe form of forgetting ("stage catastrophic forgetting") compared to adding multiple classes at once.
  • ContCap is a scalable framework: The modular nature of the proposed techniques allows for integration with different captioning architectures and future advancements.

Key Results

  • Fine-tuning severely degrades performance on old tasks: In the "addition of one class" scenario, fine-tuning resulted in a drastic drop in CIDEr score from 47.3 to 9.4 on the old task.
  • Pseudo-labeling outperforms fine-tuning: It achieved a CIDEr score of 25.3 on the old task, nearly tripling the fine-tuned model's performance.
  • Feature distillation shows strong performance on new tasks: In the "addition of multiple classes at once" scenario, feature distillation achieved a CIDEr score of 60.3 on the new tasks, surpassing fine-tuning.
  • Sequential class addition is challenging: Even with advanced techniques, performance on old tasks significantly degraded, with CIDEr scores dropping to as low as 4.8 for feature distillation.
  • Qualitative results align with quantitative findings: Visual examples demonstrate that models trained with ContCap techniques produce more relevant captions for both old and new tasks, especially when compared to simple fine-tuning.

Limitations and Future Work

  • Sequential class addition remains a challenge: The paper highlights "stage catastrophic forgetting" as a significant issue when classes are added incrementally, leading to poor performance on old tasks. Further investigation into this specific scenario is needed.
  • "Unknown-domain addition" vs. "known-domain addition": The authors suggest that experiments with "known-domain addition" (where new classes might share some characteristics with old ones) could lead to less forgetting than the "unknown-domain addition" scenario tested.
  • Multi-modal nature of image captioning: The paper acknowledges that further approaches might be needed to fully leverage the multi-modal nature of image captioning for high performance, similar to advancements in single-modal tasks.
  • Scalability for video captioning: The framework is presented as scalable for continual image or video captioning, suggesting future work could extend these techniques to video data.

Summary updated

Abstract

While advanced image captioning systems are increasingly describing images coherently and exactly, recent progress in continual learning allows deep learning models to avoid catastrophic forgetting. However, the domain where image captioning working with continual learning has not yet been explored. We define the task in which we consolidate continual learning and image captioning as continual image captioning. In this work, we propose ContCap, a framework generating captions over a series of new tasks coming, seamlessly integrating continual learning into image captioning besides addressing catastrophic forgetting. After proving forgetting in image captioning, we propose various techniques to overcome the forgetting dilemma by taking a simple fine-tuning schema as the baseline. We split MS-COCO 2014 dataset to perform experiments in class-incremental settings without revisiting dataset of previously provided tasks. Experiments show remarkable improvements in the performance on the old tasks while the figures for the new surprisingly surpass fine-tuning. Our framework also offers a scalable solution for continual image or video captioning.
BibTeX
@misc{nguyen2019contcap,
  author = {Giang Nguyen and Tae Joon Jun and Trung Tran and Daeyoung Kim},
  title = {{ContCap: A comprehensive framework for continual image captioning}},
  year = {2019},
  eprint = {1909.08745},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.08745},
}
Figure