Maximum Margin Reward Networks for Learning from Explicit and Implicit Supervision

Haoruo Peng, Ming-Wei Chang, Wen-tau YihEMNLP 2017 · OpenReview

Abstract

Structured output prediction in Natural Language Processing (NLP) typically relies on expensive, expert-labeled explicit supervision. This paper introduces Maximum Margin Reward Networks (MMRN), a unified framework that bridges the gap between explicit gold labels and implicit, delayed feedback. By casting structured prediction as a search problem and employing a max-margin loss, MMRN enables models to learn effectively from both types of supervision. The framework achieves state-of-the-art performance in Named Entity Recognition (NER) and Semantic Parsing.

The Challenge: Explicit vs. Implicit Supervision

Traditional NLP models for tasks like Named Entity Recognition (NER) or Semantic Parsing require explicit supervision, where every component of the output structure is labeled by humans. For instance, in NER, every token must be tagged as a person, location, or other entity. While effective, this approach is costly and does not scale easily.

An alternative is implicit supervision (or weak supervision), where the model only receives feedback on the final outcome. In semantic parsing, for example, we may not have the "gold" logical query for a natural language question, but we can verify if the query generated by the model retrieves the correct answer from a database.

Implicit supervision in semantic parsing: the model generates a logical form to query a knowledge base (KB), receiving feedback only on the final answer.

The primary difficulty with implicit signals is that they are delayed and noisy. If a multi-step query fails, it is unclear which specific step was incorrect. MMRN addresses this by providing a unified learning objective for both immediate and delayed rewards.

Methodology: Search-Based Structured Prediction

MMRN treats structured prediction as a search problem within a state-action space. Instead of predicting the entire structure at once, the model builds the output step-by-step.

Unified Reward Function

The framework defines a reward $R$ for every action.

  • Explicit tasks: The reward is immediate (e.g., 1 for a correct tag, 0 otherwise).
  • Implicit tasks: The reward is delayed, calculated at the end of the sequence by comparing the final output to the ground truth (e.g., using F1 score).

Max-Margin Learning

The learning objective is based on a max-margin loss, ensuring that the "best" path (the one maximizing the sum of rewards and model scores) is separated from the "most violated" path (where the model is confident but the reward is low) by a margin proportional to their rewards. This approach allows the model to learn even when the optimal path is not explicitly provided.

Neural Architectures

The authors employ task-specific neural designs to implement the scoring function $f_\theta(s, a)$. For NER, a Bi-LSTM encodes the input sequence, while the state $s$ and action $a$ (tag type) are used to compute the score for each step.

The neural architecture for NER, utilizing a Bi-LSTM to encode input and a scoring function $f_\theta(s, a)$ to guide the search for the best tag sequence.

For semantic parsing, the model uses feed-forward networks to score logical forms against a knowledge base, often updating an entity linker jointly with the parser to improve overall performance.

Experimental Results

The MMRN framework was evaluated on several benchmark datasets:

  • Named Entity Recognition: On the CoNLL-2003 dataset, MMRN achieved an F1 score of 91.39%, surpassing previous state-of-the-art models without the need for external gazetteers.
  • Semantic Parsing: On the WebQSP dataset, a joint MMRN model reached 68.1% F1, representing a 1.4% absolute improvement over the previous best system (STAGG).
  • Joint Learning: Ablation studies confirmed that training components jointly (e.g., entity linking and parsing) significantly outperforms traditional pipeline approaches.

Conclusion

Maximum Margin Reward Networks provide a robust, unified framework for learning from diverse supervision signals. By integrating search-based inference with a max-margin objective, MMRN effectively handles the noise and delay inherent in implicit feedback. Future work will focus on scaling these methods to larger search spaces and addressing the "cold start" problem in tasks with sparse rewards.

Summary updated

BibTeX
@inproceedings{peng2017maximum,
  author = {Haoruo Peng and Ming-Wei Chang and Wen-tau Yih},
  title = {{Maximum Margin Reward Networks for Learning from Explicit and Implicit Supervision}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=rJbluzGObH},
}
Figure