TL;DR
This paper investigates whether syntactic dependency relations, derived from Universal Dependencies, can improve stance detection performance across five languages. While classical machine learning models showed some benefit from these features, the state-of-the-art M-BERT model did not consistently improve, suggesting that semantic understanding might be more crucial for stance detection than syntactic structure alone.
What Problem Does This Paper Address?
Stance detection (SD) is a crucial NLP task focused on identifying a person's attitude (e.g., for, against, neutral) towards a specific target, often in political or public discourse. While similar to sentiment analysis, SD targets a specific entity or topic. Existing research has explored various linguistic features, but the specific contribution of syntactic information, particularly dependency relations, to SD performance, especially in a multilingual setting, remains an open question. The authors aim to fill this gap by systematically evaluating the impact of dependency-based features on both classical and neural models.
Methods
The researchers employed a two-pronged experimental approach:
- Classical Machine Learning with Feature Engineering: They extracted a rich set of features derived from Universal Dependencies (UD) parse trees. These included:
- N-grams and Char-grams: Standard token-level and character-level features.
- Dependency Relation Features: Features capturing the presence of negation (
deprelneg) and bags of dependency relations (deprel). - Relation-based Features: Features capturing token relationships around verbs, nouns, and adjectives (
relationformVERB,relationformNOUN,relationformADJ). -
Sidorov's Features: Bigrams based on word forms, part-of-speech tags, and dependency relations extracted from syntactic trees, rather than linear text order.
These features were then fed into several classical machine learning models: Support Vector Machine (SVM), Logistic Regression (LR), Random Forest (RF), and Multilayer Perceptron (MLP). The goal was to select the best feature combinations for each language.
-
Syntactically-Informed M-BERT: The state-of-the-art Multilingual Bidirectional Encoder Representations from Transformers (M-BERT) model was used. They experimented with:
- The base M-BERT architecture.
- M-BERT augmented with the dependency-based syntactic features described above, treated as additional input.
-
M-BERT augmented with a curated subset of the best-performing features identified in the first setting.
This setup aimed to isolate the contribution of syntactic features when integrated into a powerful pre-trained language model.
Datasets
The study utilized several publicly available datasets for stance detection across five languages:
- English: Hillary Clinton (SemEval 2016 Task 6)
- Spanish: Independence Referendum (from the research group, previously used in Lai et al., 2017)
- Catalan: Independence Referendum (StanceCat at IberEval 2017)
- French: Emmanuel Macron (from the research group, previously used in Lai et al., 2020)
- Italian:
- Constitutional Referendum (from the research group, previously used in Lai et al., 2020)
- Sardines Movement (SardiStance at EVALITA 2020)
All datasets were pre-processed using the UDPipe tool to generate Universal Dependencies parse trees.
Key Insights and Ideas
- Classical Models Benefit from Syntax: For classical ML models, dependency-based features, particularly those inspired by Sidorov et al.'s work (syntactic bigrams), consistently contributed to better performance across most language scenarios. This suggests that syntactic structure can indeed encode valuable information for stance detection when combined with other features.
- M-BERT's Limited Gain from Syntax: The integration of dependency-based features into the M-BERT model did not yield consistent or significant improvements. In many cases, the performance either stayed the same or even decreased compared to the base M-BERT architecture.
- The "Sui Generis" Italian Dataset: The Italian dataset on the Constitutional Referendum showed unusually high performance with simple features like n-grams and hashtags. The authors attribute this to the dataset's specific characteristics, where polarized hashtags are prevalent and strongly indicative of stance, making syntactic features less critical.
- Semantics Over Syntax for Stance: The findings suggest that while syntax can be helpful, especially for simpler models, the core of stance detection might lie more in semantic understanding. The ability of a model to grasp the meaning and nuances of language appears to be more critical than its understanding of grammatical structure for this task, particularly for advanced models like M-BERT.
- Identical Syntactic Structure, Opposite Stance: The authors highlight a fundamental challenge: simple sentences with identical syntactic structures can express opposite stances (e.g., "I love the Sardines Movement" vs. "I hate the Sardines Movement"), underscoring the need for semantic interpretation.
Key Results
- Classical ML: In the best runs with classical ML algorithms, at least one dependency-based syntactic feature was exploited in almost all configurations. The Multilayer Perceptron (MLP) was identified as the best-performing classical ML algorithm across most languages. For example, in the English Hillary Clinton task, MLP with selected features achieved a macro F1-score of .673.
- M-BERT: The addition of syntactic knowledge to M-BERT resulted in varied outcomes. In many cases, the performance was either unchanged (indicated by orange values in Table 3) or decreased (red values) compared to the bare M-BERT architecture. For instance, for the French Emmanuel Macron task, M-BERT+syntax achieved .511, while bare M-BERT achieved .526. However, for the Italian Constitutional Referendum, M-BERT+syntax achieved .960, a slight improvement over bare M-BERT (.959).
- Statistical Significance: The t-test with Bonferroni correction indicated that the improvements (or degradations) observed with M-BERT+syntax were often not statistically significant, further questioning the consistent benefit of these features for the neural model.
Limitations and Future Work
- Limitations:
- The authors acknowledge that the performance of their approach is heavily reliant on the quality of the dependency parsing. While UDPipe is state-of-the-art, parsing errors can still impact results.
- The study found that syntactic features, when added to M-BERT, did not provide a strong or consistent beneficial contribution, and in many cases, the results were lower than the base architecture.
- The statistical significance of the improvements observed with M-BERT was often low.
- Future Work:
- Explore new ways of encoding syntactic features and integrating them into BERT.
- Experiment with language-specific BERT architectures (e.g., AlBERTo for Italian, BETO for Spanish, CamemBERT for French) instead of the multilingual version.
- Further investigate the behavior of features when combined with pre-trained language models, aiming for a better understanding of their contribution.
- The authors suggest that the expression of stance might depend more on semantics than on syntactic patterns, implying a need for deeper semantic analysis.
Summary updated