Incorporating Unlikely Negative Cues for Distinctive Image Captioning

Zhengcong Fei, Junshi HuangIJCAI 2023 · DOI

TL;DR

This paper introduces "Unlikely Negative Knowledge Training" (UNKT), a novel approach to improve image captioning by teaching models what not to generate. By training a "negative teacher" model to produce generic captions and then using this to guide a "student" model, UNKT significantly enhances caption diversity and accuracy, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Neural image captioning models have made impressive strides, but they often struggle with generating generic and repetitive descriptions. This "many-to-one" phenomenon, where a single image can have multiple valid captions but models tend to produce safe, common ones, limits their usefulness in real-world applications. While previous methods have tried to boost diversity, they often focus on high-frequency or common tokens as negative examples. This paper argues that this overlooks low-frequency but still generic and meaningless sentences, which can also hinder caption quality. The core problem is how to effectively guide captioning models to produce more distinctive, creative, and contextually rich descriptions by learning from what makes a caption "bad" or generic.

Methods

The proposed method, Unlikely Negative Knowledge Training (UNKT), employs a teacher-student framework.

  • Negative Teacher Construction: A "negative teacher" model is trained on a specially curated dataset. This dataset is created using a "retrieval entropy filtering strategy" that identifies image-text pairs likely to correspond to generic captions (i.e., many images associated with similar text). The teacher model, trained with Maximum Likelihood Estimation (MLE) on this data, naturally learns to produce generic captions.
  • Unlikely Negative Knowledge Transfer: The core innovation lies in how knowledge is transferred from the negative teacher to the student model. Instead of traditional knowledge distillation (where the student mimics the teacher), UNKT uses the teacher's generic outputs to penalize the student. This transfer happens at multiple levels:
  • Prediction Layer: A "soft unlikelihood loss" is introduced to maximize the distance between the student's predicted probabilities and the teacher's generic predictions. This goes beyond simple token-level penalties by considering the entire probability distribution.
  • Intermediate Layers: Implicit knowledge within the student's hidden states and attention matrices is also leveraged. A "mean Reverse KL divergence" (RKL) is used to increase the distance between the teacher's and student's intermediate representations.
  • Progressive Hyper-Parameter Adjustment: To ensure the student model can generate captions before being penalized, a progressive approach is used for the negative knowledge weight (λ). It starts low and gradually increases, then decreases, resembling cosine learning rate decay.

Datasets

The primary dataset used for training and evaluation is MS COCO. The paper uses the standard m-RNN split for training, validation, and testing. No new benchmarks were introduced.

Key Insights and Ideas

  • Learning "What Not to Say": The most significant insight is the effectiveness of explicitly teaching models what constitutes a "bad" or generic caption, rather than just encouraging "good" ones. This "unlikely negative knowledge" is crucial for breaking free from generic outputs.
  • Multi-Level Knowledge Transfer: Transferring negative knowledge not just from the final prediction layer but also from intermediate hidden states and attention mechanisms provides a richer signal for discouraging undesirable behaviors.
  • Retrieval Entropy for Generic Captions: Using retrieval entropy as a proxy for identifying generic captions is an effective way to curate a targeted negative training set, going beyond simple frequency-based filtering.
  • Teacher-Student for Negative Guidance: The teacher-student paradigm, repurposed for negative guidance, is a flexible and powerful framework for this task.

Key Results

  • Quality Improvement: UNKT significantly improves caption quality metrics (BLEU, METEOR, ROUGE-L, CIDEr, SPICE) compared to state-of-the-art baselines, especially in the oracle setting. For example, UNKT achieved a CIDEr score of 174.2 with 20 samples, outperforming previous methods.
  • Diversity Enhancement: The method also shows substantial gains in diversity metrics (SelfCIDEr, mBLEU, Div-n), demonstrating its ability to generate a wider range of distinct captions for the same image.
  • Ablation Studies: Removing any of the UNKT components (prediction layer loss, hidden state transfer, attention transfer) led to performance degradation, highlighting the importance of each element. The hidden state transfer was particularly crucial.
  • Human Evaluation: Human evaluators rated UNKT captions as more informative and relevant than those from baselines, confirming the automatic metric results.

Limitations and Future Work

  • Memory Degradation on Diversity: The paper notes that incorporating a "memory" component (likely referring to advanced memory networks) slightly degraded diversity performance, suggesting a trade-off or need for further tuning.
  • Generalizability: While the paper demonstrates UNKT's applicability to Transformer architectures and mentions its potential for other advanced models, further exploration across a wider range of architectures could be beneficial.
  • Human Evaluation Nuances: While UNKT performed well, the human evaluation results show that baselines still achieved competitive scores in some aspects (e.g., fluency), indicating room for further refinement.

The authors suggest that UNKT is a general paradigm that can be easily applied to existing image captioning models to improve their diversity, implying future work could focus on integrating it into even more sophisticated architectures.

Summary updated

BibTeX
@inproceedings{fei2023incorporating,
  author = {Zhengcong Fei and Junshi Huang},
  title = {{Incorporating Unlikely Negative Cues for Distinctive Image Captioning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/83},
}
Figure