TL;DR
This paper tackles the challenge of zero-shot and few-shot stance detection, where models need to identify opinions on topics with little to no training data. They introduce a novel approach that leverages commonsense knowledge graphs to bridge the gap between documents and topics, significantly improving performance on this difficult task.
What Problem Does This Paper Address?
Stance detection, the task of identifying an author's attitude (pro, con, neutral) towards a specific topic, has traditionally relied on ample labeled data for each target topic. However, this approach is impractical when dealing with a vast number of new topics, as manually annotating data is time-consuming and expensive. "Cross-target stance detection" attempts to generalize from related topics, but still requires significant training data.
This paper focuses on the more challenging zero-shot and few-shot stance detection scenarios. The core problems are:
- Generalization: Models struggle to perform well on topics they haven't seen during training, or have only seen a few examples of.
- Implicit Stance: Topics are often not explicitly mentioned in the text, making it difficult to connect the document's content to the target topic. This is illustrated in Figure 1, where the topic "Stability" is not directly present in the text.
Traditional data-driven methods fall short in these scenarios. The authors propose that commonsense knowledge, which humans use effortlessly for reasoning, can be a crucial missing piece to enhance model generalization and reasoning capabilities.
Methods
The proposed method, CKE-Net (Commonsense Knowledge Enhanced Network), integrates commonsense knowledge from an external knowledge graph (ConceptNet) into a BERT-based stance detection framework.
Here's a breakdown of their approach:
- BERT Encoding: They use a pre-trained BERT model to encode both the document and the target topic. The input is formatted as
[CLS] document [SEP] topic [SEP]. This provides rich contextual representations of the text and topic. - Commonsense Knowledge Graph Extraction: They leverage ConceptNet, a large knowledge graph containing millions of relation triples. To connect a document and its topic, they extract a relational subgraph. This subgraph is built by finding two-hop directed paths between concepts mentioned in the document and concepts related to the topic. Importantly, they also add reverse edges between concept pairs to facilitate information flow.
- Graph Encoding with CompGCN: To effectively utilize the relational information within the subgraph, they employ CompGCN (Composition-based Graph Convolutional Network). Unlike standard GCNs that focus on node information, CompGCN jointly embeds both nodes and relations. This allows the model to learn from the structure and semantics of the knowledge graph.
- Stance Classification: Finally, the representations from the BERT encoder (document and topic) are concatenated with the relational representations derived from the knowledge graph. This combined representation is fed into a two-layer Multi-Layer Perceptron (MLP) with a softmax function to predict the stance label. The model is trained using a multi-class cross-entropy loss.
Datasets
The primary dataset used for evaluation is Varied Stance Topics (VAST), introduced by Allaway and McKeown (2020). VAST is specifically designed for zero-shot and few-shot stance detection and contains thousands of topics. A key characteristic of VAST is that documents in the training set do not appear in the validation or test sets, and zero-shot topics in the test set are entirely absent from training.
The paper also briefly mentions other stance detection datasets (e.g., Mohammad et al., Kobbe et al., Vamvas and Sennrich) but notes they have fewer topics, making them less suitable for evaluating zero-shot and few-shot capabilities.
Key Insights and Ideas
- Commonsense is Crucial for Generalization: The paper strongly suggests that commonsense knowledge is not just a "nice-to-have" but a fundamental requirement for models to generalize to unseen topics in stance detection.
- Relational Knowledge is Key: Simply using word-level semantic knowledge (as some prior work did) is insufficient. The relationships between concepts in a knowledge graph are vital for bridging the semantic gap between a document and its target topic, especially when the topic is implicit.
- Bridging the Implicit Gap: The example in Figure 1 highlights how commonsense relations (like "stability" being related to "job" and "tenure") can help infer stance even when the topic isn't explicitly stated.
- CompGCN for Relational Graphs: The choice of CompGCN is significant because it's designed to handle graphs with explicit relations, making it a natural fit for leveraging structured knowledge graphs like ConceptNet.
Key Results
The proposed CKE-Net model consistently outperforms state-of-the-art baselines across various scenarios:
- Overall Performance: CKE-Net achieves a significant improvement in macro-averaged F1-score on the VAST test set for both zero-shot and few-shot tasks, demonstrating its superior generalization ability.
- Zero-Shot vs. Few-Shot: The model shows substantial gains on both zero-shot and few-shot topics, indicating its effectiveness in scenarios with minimal or no prior exposure to the target topic.
- Challenging Phenomena: CKE-Net excels in handling difficult cases, including:
- Imp (Implicit Topic): Demonstrates that relational knowledge helps understand the topic-document relationship when the topic is not mentioned.
- Qte (Quotations) and Sarc (Sarcasm): Shows improved performance on texts employing special rhetorics, likely due to the richer semantic understanding provided by the knowledge graph.
- BERT-based Models: The paper notes that BERT-based models generally perform better than older BILSTM-based methods, and CKE-Net further enhances these BERT models.
Limitations and Future Work
- Reliance on External KG: The performance of CKE-Net is inherently dependent on the quality and coverage of the external knowledge graph (ConceptNet). If a topic or its relevant concepts are not well-represented in ConceptNet, the model's performance might be limited.
- Computational Cost: Integrating graph encoding with BERT can increase computational complexity compared to pure text-based models.
- Future Work: The authors suggest that further exploration of how to better integrate relational knowledge and potentially more advanced graph neural network architectures could lead to even greater improvements. They also imply that applying this approach to other NLP tasks requiring reasoning and generalization would be a valuable direction.
Summary updated