TL;DR
This paper tackles the challenge of identifying subjectivity in arguments about human values, where annotator disagreement is common. They propose two main approaches: inferring subjectivity from individual value predictions and directly classifying subjectivity. Their findings show that directly identifying subjectivity significantly outperforms inferring it, offering a more robust way to flag arguments that might be interpreted differently.
What Problem Does This Paper Address?
Human values, like benevolence or self-determination, are fundamental to our motivations and guide our actions. Arguments grounded in these values are called "value-laden arguments." Identifying the specific values that underpin these arguments is a growing area of interest in NLP. However, the interpretation of values can be highly subjective, leading to natural disagreement among human annotators. Current methods often rely on majority aggregation of annotations, which can mask these valuable insights into subjective interpretations. This paper aims to address this by developing methods to automatically identify when annotators are likely to disagree on the values associated with an argument, thus enabling a more nuanced understanding of subjectivity in this domain.
Methods
The authors propose two main approaches:
- Inferred Subjectivity Identification (IS): This approach involves training models to predict value labels for individual annotators and then inferring subjectivity based on the variations in these predictions. They explore three architectures:
- IS-each: A separate model for each annotator.
- IS-shared: A single model with shared embeddings but a dedicated classification head for each annotator.
- IS-single: A single model for all annotators, incorporating the annotator ID as an input feature.
- Direct Subjectivity Identification (DS): This approach directly trains a model to classify whether an argument is subjective (i.e., likely to cause annotator disagreement). They compare three methods:
- DS-simple: A dedicated binary classification model for each value label.
- DS-sup: DS-simple enhanced with a supervised contrastive loss (triplet loss) to push embeddings of similar (subjective) and dissimilar (non-subjective) examples apart.
- DS-unsup: DS-simple enhanced with an unsupervised contrastive loss (contrastive tension loss) to bring semantically similar samples closer together.
Datasets
The primary dataset used is the Touché23-ValueEval dataset. This dataset contains natural language arguments annotated with a taxonomy of 54 values, derived from the Schwartz Value Survey. Crucially, it includes annotations from multiple annotators for each argument, allowing for the study of inter-annotator agreement and subjectivity. The authors selected the eight most frequently annotated values from level 2 for their experiments.
Key Insights and Ideas
- Direct is Better: Directly identifying subjectivity (DS approach) significantly outperforms inferring it from value predictions (IS approach). This suggests that focusing the model's objective on subjectivity itself leads to better performance.
- Subjectivity Matters for Performance: For the IS approach, performance is correlated with the subjectivity of the value annotations. However, for the best-performing DS methods (DS-simple and DS-sup), lower subjectivity (less disagreement) leads to better performance. This implies that subjective arguments are inherently harder to classify.
- Contrastive Learning's Role: While contrastive loss didn't directly improve subjectivity prediction performance in all cases, it did lead to models that were less dependent on per-label subjectivity. Visualizations show that contrastive learning helps create more separable embedding spaces for subjective and non-subjective arguments.
- Model Agnosticism: The proposed methods are designed to be model-agnostic, allowing them to be applied with different underlying embedding models.
Key Results
- The DS approach consistently outperforms the IS approach across various metrics, including precision, recall, and F1-score. Even a random baseline often outperformed the IS methods, except for specific cases like recall for "Ach" (Achievement).
- Among the IS methods, IS-each (dedicated model per annotator) performed best, suggesting that capturing individual annotator tendencies is beneficial, though computationally expensive.
- For the DS methods, DS-simple and DS-sup generally performed similarly and better than DS-unsup.
- The paper reports Spearman correlation scores between per-value F1-scores and subjectivity ratios. For IS methods, a moderate positive correlation was observed. For DS methods, a negative correlation was found for DS-simple and DS-unsup, indicating better performance on less subjective values. DS-sup showed a more balanced performance across subjectivity levels.
Limitations and Future Work
- Dataset Scarcity: The evaluation is limited to the Touché23-ValueEval dataset due to the scarcity of datasets with annotator-level value annotations. The authors emphasize the need for more diverse datasets to facilitate robust evaluation.
- Computational Cost: The IS-each and IS-shared methods are computationally expensive due to dedicated embeddings and classifiers. Future work could focus on optimizing these for scalability or developing lightweight alternatives.
- Classification Focus: The study is limited to classification tasks. Future work could explore how subjectivity manifests in other NLP tasks like summarization or question-answering.
- Confounding Factors: The authors acknowledge that annotation differences (subjectivity) can be confounded with noise. Future work could explore methods that explicitly model the relationship between value prediction and subjectivity identification, potentially by incorporating annotator demographics or other contextual factors.
- Context-Specific Values: Values are context-specific, and the current models might need fine-tuning for new contexts or deliberation settings.
- Combination of Approaches: The authors envision combining value and subjectivity prediction to capture individual annotator perspectives more effectively.
Summary updated