Introduction
In the field of Natural Language Generation (NLG), the attention mechanism serves as a critical bridge, aligning specific parts of an input sequence with the generated output. While models typically learn these alignments autonomously, "supervised attention" attempts to guide this process using external knowledge. However, obtaining precise, one-to-one alignments is often prohibitively expensive or theoretically impossible for complex structures like knowledge graphs. This paper introduces Generalized Supervised Attention (GSA), a framework designed to leverage "quasi-alignments"—sets of potential source candidates—to improve model performance without requiring perfect supervision.
The Challenge of Precise Alignment
Standard supervised attention relies on hard targets, assuming that for every output token, there is exactly one "correct" source token. In reality, a single word might relate to multiple nodes in a graph, or the supervision signal itself might be noisy. When forced to follow ambiguous or slightly incorrect alignments, traditional models often perform worse than their unsupervised counterparts. GSA addresses this by allowing the model to learn from a "candidate set" of relevant items rather than a single point of truth.
Methodology: SCE Loss and SMA Architecture
The GSA framework introduces two primary technical innovations to handle alignment ambiguity:
Summation Cross-Entropy (SCE) Loss
Traditional Cross-Entropy (CE) loss encourages a model to distribute attention uniformly across all supervised targets. If a word is aligned to three nodes, CE forces the model to give each 33% attention. In contrast, Summation Cross-Entropy (SCE) maximizes the total sum of attention weights across the candidate set. This flexibility allows the model to dynamically identify the most relevant item within the set during training, effectively "filtering" the quasi-alignment.
Supervised Multiple Attention (SMA)
To balance human guidance with data-driven discovery, the authors propose Supervised Multiple Attention (SMA). This architecture, inspired by multi-head attention, dedicates one attention channel to follow the supervised quasi-alignments while allowing other channels to remain unsupervised. This hybrid approach ensures the model benefits from known structural constraints while retaining the ability to capture latent patterns the human-provided rules might overlook.
Experimental Evaluation
The researchers evaluated GSA across three distinct NLG tasks:
- Data-to-Text: Generating scientific abstracts from knowledge graphs (AGENDA dataset).
- AMR-to-Text: Converting Abstract Meaning Representation graphs into sentences (LDC2015E86 dataset).
- Text Summarization: Generating summaries for the CNN/DailyMail dataset.
Key Results
GSA consistently outperformed unsupervised baselines and traditional supervised methods. In the Data-to-Text task, the model achieved 15.51 BLEU, a significant improvement over the 14.30 unsupervised baseline. Similar gains were observed in AMR-to-Text (improving from 28.39 to 29.30 BLEU). While gains in summarization were more modest, the authors attribute this to the high natural alignment coverage already present in standard summarization models.
Discussion: Robustness and Qualitative Analysis
A core strength of GSA is its resilience to noise. Experiments demonstrated that the model maintains superior performance even when up to 20% of the supervision signals are erroneous. Qualitatively, the SCE loss proves superior at resolving ambiguity. For instance, when an input contains multiple instances of the same word (e.g., "police"), the SCE-trained model uses surrounding context to focus on the correct instance, whereas CE-trained models often become confused and blur their attention.
Conclusion
Generalized Supervised Attention provides a robust framework for incorporating "fuzzy" human knowledge into text generation models. By shifting from rigid, one-to-one supervision to flexible quasi-alignments via SCE loss and SMA architecture, GSA overcomes the bottleneck of perfect data labeling. Future work may extend this approach to multi-modal domains, such as image captioning, where spatial alignments are inherently ambiguous.
Summary updated