Teaching Machines to Describe Images with Natural Language Feedback

Huan Ling, Sanja FidlerNeurIPS 2017 · OpenReview

Reinforcement Learning (RL) has become a standard paradigm for training AI agents, but it typically relies on scalar rewards—a single number representing "good" or "bad." In complex tasks like image captioning, a numeric reward is effectively "blind." It tells the model that a generated sentence is poor, but it fails to explain why or where the mistake occurred. This limitation is known as the credit assignment problem.

To address this, researchers propose that natural language is the ideal medium for non-expert users to guide machines. Just as a parent corrects a child by pointing out specific errors, a human teacher can provide a descriptive sentence that identifies a mistake and suggests a correction. The challenge, however, lies in translating this unstructured text feedback into a mathematical signal the model can actually use to improve.

Example of natural language feedback correcting a machine's caption.

A Framework for Language-Guided Learning

To bridge the gap between human speech and machine learning, the authors developed a novel three-part framework:

  1. Hierarchical Phrase-based Captioner: Unlike standard models that generate captions word-by-word, this model uses a two-level Recurrent Neural Network (RNN). A "phrase-level" RNN decides the topic and type of phrase (e.g., Noun Phrase, Verb Phrase), while a "word-level" RNN generates the specific words. This hierarchical structure makes it significantly easier to map human feedback to specific segments of the sentence.
  2. The Feedback Network (FBN): This is a separate neural network trained to "understand" the human critique. It takes a generated caption and the human's natural language feedback as input, then classifies each phrase in the caption as correct, wrong, or not relevant.
  3. Policy Gradient RL with Feedback: The model is trained using the REINFORCE algorithm with an augmented reward function. Instead of relying solely on automated metrics like BLEU scores, the model receives a positive reward for phrases the FBN deems "correct" and a negative reward for those deemed "wrong" based on the human's text.

Data Collection and Key Insights

The researchers evaluated their approach using the standard MS-COCO benchmark and a newly collected Human Feedback Dataset. Using Amazon Mechanical Turk, they gathered detailed feedback on 9,000 images, including the type of mistake, the specific area of the error, and a natural language explanation.

The study revealed several critical insights:

  • Language as a Stronger Signal: Descriptive feedback proved to be highly sample-efficient. The model performed better when given descriptive feedback on new images than when it was simply provided with additional, perfect human-written captions for those same images.
  • Solving Credit Assignment: By pointing to a specific phrase (e.g., "a cat") and providing a reason ("There is a dog on a sidewalk, not a cat"), the feedback allows the model to update specific parts of its policy rather than blindly guessing which word in a sentence was problematic.
  • Linguistic Priors Matter: The model performs significantly better when it explicitly reasons about phrase types. This allows the attention mechanism to focus on objects when generating nouns and global context when generating verbs.

Results and Future Directions

The inclusion of natural language feedback yielded strong empirical results. In head-to-head evaluations, humans preferred the captions from the RL-with-feedback model 4.7% more often than those from a baseline RL model. Quantitatively, the model achieved a "Weighted Metric" (combining BLEU and ROUGE) of 107.67, outperforming both the baseline RL model (106.55) and a standard Maximum Likelihood Estimation model (104.12). Furthermore, the method required nearly 50% fewer clicks from human annotators to achieve better results compared to traditional methods of providing full ground-truth captions.

While promising, the current system relies on static, pre-collected feedback and is bottlenecked by the FBN's accuracy (currently around 74.6%). A misinterpreting FBN can provide noisy rewards that confuse the learner. Future work will need to explore real-time, interactive conversations between humans and agents, enabling active learning where the model can ask for clarification when uncertain.

Summary updated

Abstract

Robots will eventually be part of every household. It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. A descriptive sentence can provide a stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the problem of image captioning in which the quality of the output can easily be judged by non-experts. We propose a phrase-based captioning model trained with policy gradients, and design a critic that provides reward to the learner by conditioning on the human-provided feedback. We show that by exploiting descriptive feedback our model learns to perform better than when given independently written human captions.
BibTeX
@inproceedings{ling2017teaching,
  author = {Huan Ling and Sanja Fidler},
  title = {{Teaching Machines to Describe Images with Natural Language Feedback}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=S1ZB2I-dbS},
}
Figure