Abstract
Attention is typically used to select informative sub-phrases that are used for prediction. This paper investigates the novel use of attention as a form of feature augmentation, i.e., casted attention. We propose Multi-Cast Attention Networks (MCAN), a new attention mechanism and general model architecture for a potpourri of ranking tasks in the conversational modeling and question answering domains. Our approach performs a series of soft attention operations, each time casting a scalar feature upon the inner word embeddings. The key idea is to provide a real-valued hint (feature) to a subsequent encoder layer and is targeted at improving the representation learning process. There are several advantages to this design, e.g., it allows an arbitrary number of attention mechanisms to be casted, allowing for multiple attention types (e.g., co-attention, intra-attention) and attention variants (e.g., alignment-pooling, max-pooling, mean-pooling) to be executed simultaneously. This not only eliminates the costly need to tune the nature of the co-attention layer, but also provides greater extents of explainability to practitioners. Via extensive experiments on four well-known benchmark datasets, we show that MCAN achieves state-of-the-art performance. On the Ubuntu Dialogue Corpus, MCAN outperforms existing state-of-the-art models by 9%. MCAN also achieves the best performing score to date on the well-studied TrecQA dataset.
Introduction
In retrieval-based Question Answering (QA) and dialogue systems, the primary challenge is to rank candidate responses based on their relevance to a given query. While attention mechanisms have become a staple in these systems, they are traditionally used to "pool" or "select" information at the end of a processing pipeline. This paper introduces Multi-Cast Attention Networks (MCAN), which reimagines attention as a form of feature augmentation. Instead of a final selection step, attention is used to generate "hints" that enrich word embeddings before they reach the primary encoder.
Methodology
The core innovation of MCAN is the concept of Casted Attention. Standard attention methods often lead to "representation bloat" because concatenating multiple high-dimensional attention vectors significantly increases the parameter count of subsequent layers. MCAN bypasses this by compressing attention signals into scalar values.
The MCAN Pipeline
- Highway Encoder: Raw word embeddings are first processed through a highway layer, which acts as a data-driven filter to control information flow.
- Multi-Cast Attention: The model simultaneously executes multiple attention operations. This includes Co-Attention (comparing query to document using Max, Mean, and Alignment pooling) and Intra-Attention (self-attention within a single sentence to capture long-term dependencies).
- Compression: Comparison features (concatenation, multiplication, and subtraction) are generated for each attention type and then compressed into a single scalar "hint" per word using functions like Sum, Neural Networks, or Factorization Machines.
- LSTM Encoder: These scalar hints are appended to the original word embeddings. A Siamese LSTM then processes these augmented embeddings, leveraging the global and cross-sentence context provided by the hints.
- Prediction: Final representations are pooled using a MeanMax strategy and passed through a final highway network to generate a similarity score.
Experimental Evaluation
The authors evaluated MCAN on four diverse benchmark datasets: the Ubuntu Dialogue Corpus (UDC) for technical chat prediction, TrecQA for factoid QA, QatarLiving for community forum threads, and a Tweet Reply Prediction dataset.
Key Results
MCAN achieved state-of-the-art (SOTA) results across all benchmarks at the time of publication:
- Ubuntu Dialogue Corpus: MCAN reached an R10@1 score of 0.551, outperforming the previous SOTA by 9%.
- TrecQA: The model achieved a MAP of 0.838 and an MRR of 0.904 on the "clean" setting, the highest scores recorded at that time.
- Ablation Studies: Removing the Multi-Cast Attention (MCA) mechanism resulted in the most significant performance degradation, confirming that the scalar hints are the primary drivers of the model's success. Alignment-based attention was identified as the most critical individual component.
Discussion and Analysis
A major advantage of the MCAN architecture is its inherent explainability. By visualizing the casted scalar features, researchers can understand what the model is "looking at" during the encoding process.
Qualitative analysis revealed that the model learns specific linguistic patterns:
- Neural Word Overlap: Feature $f_1$ was found to effectively learn a neural version of word overlap, showing distinct spikes when words appeared in both the query and the answer.
- Evidence vs. Anti-Evidence: The model successfully differentiates between supporting evidence and "anti-evidence" (words that indicate a mismatch despite high overlap elsewhere).
Conclusion
MCAN demonstrates that compressing complex attention signals into simple scalar hints can significantly improve representation learning without the computational cost of traditional concatenation. While the rise of Transformer-based architectures has changed the NLP landscape, the principle of "casting" attention as feature augmentation remains a powerful concept for efficient and explainable deep learning.
Summary updated