Introduction
Stance Detection (SD) aims to determine an author's position—pro, con, or neutral—toward a specific target. Unlike sentiment analysis, SD often requires external knowledge because the target may not be explicitly mentioned in the text. This task becomes particularly challenging in cross-target scenarios, where models must generalize to topics not seen during training. Current models often fail here by relying on "spurious correlations"—memorizing target-specific keywords rather than understanding the underlying argument.
The INJECT Framework
To address these challenges, researchers developed INJECT, a dual-encoder architecture designed to robustly integrate external knowledge. Unlike previous approaches that treat knowledge bases as complex graphs, INJECT converts all contextual information into natural language text, making it a "plug-and-play" solution for standard Transformer models like BERT, RoBERTa, and ELECTRA.
The Inject Block
The core innovation is the Inject Block. Instead of simply concatenating context to the input string, INJECT uses parallel encoders for the input (text + target) and the context. At specific layers, cross-attention allows the input encoder to "look" at the context representations. This mechanism enables the model to dynamically re-weigh the importance of tokens based on the provided context, acting as a regularizer that discourages the memorization of trigger words.
Experimental Setup
The model was evaluated on a comprehensive benchmark of 16 English SD datasets, spanning social media (e.g., semeval2016), debate forums (perspectrum), and news/fact-checking (snopes). Context was pulled from three heterogeneous sources:
- ConceptNet: Graph triplets converted into sentences.
- CauseNet: Extracted causal relations.
- LLM Prompting: Definitions and explanations generated by the T0pp model.
Key Results and Analysis
INJECT demonstrated superior performance and robustness across the benchmark, outperforming the BERT+Target baseline on 13 out of 16 datasets. On average, it provided a +1.1pp boost across various base models and achieved a massive +11.7 point increase in macro-F1 on the rumor dataset.
A critical finding was INJECT's robustness to noise. While standard models that concatenate context to input saw performance plummet when fed random sentences, INJECT maintained its accuracy. This suggests that the cross-attention mechanism effectively filters irrelevant information, focusing only on context that aids in stance determination.
Limitations and Future Work
Despite its success, the dual-encoder architecture effectively doubles the parameter count, increasing computational costs for training and inference. Additionally, the quality of LLM-generated context is sensitive to prompt wording and the risk of hallucinations. Future research may explore more parameter-efficient implementations, such as using Adapters, to achieve similar context injection without the overhead of a full second encoder.
Summary updated