Neural attention mechanisms are designed to identify the most relevant parts of an input, but they often struggle with "noise"—spurious correlations in training data that lead the model to focus on semantically irrelevant tokens. To address this, a growing body of research proposes supervised attention, where external linguistic resources like sentiment lexicons, dictionaries, and discourse markers are used to explicitly guide or regularize a model's internal weighting mechanism.
The Intuition: Attention as a Supervised Signal
In a standard Transformer or LSTM, attention weights are learned implicitly through the backpropagation of the final task loss (e.g., cross-entropy). However, as Yicheng Zou and Xuanjing Huang argue in A Lexicon-Based Supervised Attention Model for Neural Sentiment Analysis, this often results in a gap between what the model attends to and what humans consider semantically important.
The core idea of supervised attention is to treat the attention distribution itself as a target for optimization. By using a sentiment lexicon to identify "gold" tokens (words known to carry emotional weight), researchers can define a target attention distribution. The model is then trained not just to predict the correct label, but to ensure its attention weights align with these linguistic priors.
Mechanisms of Guidance
Researchers have developed several distinct ways to inject this external knowledge into the attention loop:
1. Lexicon-Based Regularization
Instead of forcing a hard constraint, many models use "soft" regularizers. Chuang Fan and Lin Gui demonstrate this in A Knowledge Regularized Hierarchical Approach for Emotion Cause Analysis, where they apply a hinge loss to penalize the model if it assigns low attention to words found in a sentiment dictionary. This ensures the model prioritizes semantically significant words while remaining flexible enough to learn context-specific terms not present in the lexicon. Similarly, Jiyue Jiang uses a knowledge-constrained attention loss in A Cognitive Stimulation Dialogue System to ensure dialogue models focus on high-intensity emotional keywords.
2. External Knowledge as Queries
Standard attention typically uses internal hidden states as queries. However, Yujiu Yang and Min Yang propose a "best of both worlds" approach in Sentiment Lexicon Enhanced Attention-Based LSTM. They derive a query vector directly from external domain-specific lexicons. This explicitly biases the model’s focus toward regions of the input that align with pre-defined emotional markers, effectively "anchoring" the attention mechanism in established linguistic categories.
3. Saliency Vectors and Static Priors
For tasks like semantic parsing or cross-lingual retrieval, the guidance can be even more direct. In A Unified Framework for Frame-semantic Parsing Based on Marker Attention, Xuefeng Su and Ru Li† introduce "saliency vectors"—binary vectors that flag potential semantic candidates based on a reference database. These vectors act as a soft prior, guiding the attention mechanism toward relevant segments of the input. In the context of information retrieval, Zhiqi Huang and James Allan use a static matrix of known translation pairs to complement learned attention, ensuring the model prioritizes literal word-level correspondences in Mixed Attention Transformer.
Why It Works: Reducing the Search Space
The primary benefit of these methods is robustness. By penalizing the neglect of high-sentiment or task-relevant terms, models learn to identify long-distance dependencies that might be missed by purely structural analysis. Cuiyun Gao shows in Affective Knowledge Enhanced Multiple-Graph Fusion Networks that supervising attention with SenticNet (an affective knowledge base) helps the model bridge the gap between complex sentence structures and the specific emotions tied to different aspects.
Furthermore, this approach improves interpretability. When attention is supervised by a lexicon, the resulting weights are more likely to align with human intuition. As noted by Kaisong Song and Wei Gao in their work on Evidence Aware Neural Pornographic Text Identification, even noisy, automatically generated lexicons provide a valuable prior that helps the model focus on indicative terms within complex semantic structures, making the model's "reasoning" cleaner and more logical.
By integrating established linguistic knowledge directly into the attention mechanism—rather than just appending it as an input feature—these models achieve a more stable and semantically grounded understanding of text.