Generalized Supervised Attention for Text Generation

Yixian Liu, Liwen Zhang, Xinyu Zhang, Yong Jiang, Yue Zhang, Kewei Tu

Introduction

In the field of Natural Language Generation (NLG), the attention mechanism serves as a critical bridge, aligning specific parts of an input sequence with the generated output. While models typically learn these alignments autonomously, "supervised attention" attempts to guide this process using external knowledge. However, obtaining precise, one-to-one alignments is often prohibitively expensive or theoretically impossible for complex structures like knowledge graphs. This paper introduces Generalized Supervised Attention (GSA), a framework designed to leverage "quasi-alignments"—sets of potential source candidates—to improve model performance without requiring perfect supervision.

The Challenge of Precise Alignment

Standard supervised attention relies on hard targets, assuming that for every output token, there is exactly one "correct" source token. In reality, a single word might relate to multiple nodes in a graph, or the supervision signal itself might be noisy. When forced to follow ambiguous or slightly incorrect alignments, traditional models often perform worse than their unsupervised counterparts. GSA addresses this by allowing the model to learn from a "candidate set" of relevant items rather than a single point of truth.

Methodology: SCE Loss and SMA Architecture

The GSA framework introduces two primary technical innovations to handle alignment ambiguity:

Summation Cross-Entropy (SCE) Loss

Traditional Cross-Entropy (CE) loss encourages a model to distribute attention uniformly across all supervised targets. If a word is aligned to three nodes, CE forces the model to give each 33% attention. In contrast, Summation Cross-Entropy (SCE) maximizes the total sum of attention weights across the candidate set. This flexibility allows the model to dynamically identify the most relevant item within the set during training, effectively "filtering" the quasi-alignment.

Supervised Multiple Attention (SMA)

To balance human guidance with data-driven discovery, the authors propose Supervised Multiple Attention (SMA). This architecture, inspired by multi-head attention, dedicates one attention channel to follow the supervised quasi-alignments while allowing other channels to remain unsupervised. This hybrid approach ensures the model benefits from known structural constraints while retaining the ability to capture latent patterns the human-provided rules might overlook.

The GSA architecture featuring the Aligner, SCE loss, and the SMA decoder structure.

Experimental Evaluation

The researchers evaluated GSA across three distinct NLG tasks:

  1. Data-to-Text: Generating scientific abstracts from knowledge graphs (AGENDA dataset).
  2. AMR-to-Text: Converting Abstract Meaning Representation graphs into sentences (LDC2015E86 dataset).
  3. Text Summarization: Generating summaries for the CNN/DailyMail dataset.

Key Results

GSA consistently outperformed unsupervised baselines and traditional supervised methods. In the Data-to-Text task, the model achieved 15.51 BLEU, a significant improvement over the 14.30 unsupervised baseline. Similar gains were observed in AMR-to-Text (improving from 28.39 to 29.30 BLEU). While gains in summarization were more modest, the authors attribute this to the high natural alignment coverage already present in standard summarization models.

Discussion: Robustness and Qualitative Analysis

A core strength of GSA is its resilience to noise. Experiments demonstrated that the model maintains superior performance even when up to 20% of the supervision signals are erroneous. Qualitatively, the SCE loss proves superior at resolving ambiguity. For instance, when an input contains multiple instances of the same word (e.g., "police"), the SCE-trained model uses surrounding context to focus on the correct instance, whereas CE-trained models often become confused and blur their attention.

Comparison of attention heatmaps: SCE (bottom) shows sharper, more accurate focus on specific tokens compared to the standard CE approach (top).

Conclusion

Generalized Supervised Attention provides a robust framework for incorporating "fuzzy" human knowledge into text generation models. By shifting from rigid, one-to-one supervision to flexible quasi-alignments via SCE loss and SMA architecture, GSA overcomes the bottleneck of perfect data labeling. Future work may extend this approach to multi-modal domains, such as image captioning, where spatial alignments are inherently ambiguous.

Summary updated

BibTeX
@inproceedings{liu2021generalized,
  author = {Yixian Liu and Liwen Zhang and Xinyu Zhang and Yong Jiang and Yue Zhang and Kewei Tu},
  title = {{Generalized Supervised Attention for Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.442},
}
Figure