What Does Perspectivism Mean? An Ethical and Methodological Countercriticism

Mathieu ValetteCOLING 2024

TL;DR

This paper critiques the "perspectivist" approach in NLP, which emphasizes annotator subjectivity and variation. It argues that while aiming for ethical inclusivity, it risks devaluing annotation work and conflating sincerity with truth. The authors propose a shift from "annotating data" to "constructing corpora" through scientific analysis, reclassifying annotation as intellectual work.

What Problem Does This Paper Address?

The paper tackles the growing trend in Natural Language Processing (NLP) towards "perspectivism" in data annotation. Traditionally, NLP aimed for objective, standardized data. However, perspectivism, influenced by postmodern thought, argues that human interpretation is inherently subjective and that variations in annotation (perspectives) are valuable information, not just noise. This approach aims to address ethical concerns about under-representation and bias in datasets. The authors question whether this paradigm truly achieves its ethical goals and whether it overlooks the fundamental nature of annotation work itself, which they liken to industrial "scientific management" (Taylorism). They highlight a potential ethical and methodological disconnect, suggesting that perspectivism, while attempting to be inclusive, might inadvertently devalue the labor of annotators and misinterpret the concept of "truth" in data.

Methods

The paper doesn't propose new algorithms or models. Instead, it offers a critical analysis and a conceptual re-framing of data annotation in NLP. The authors employ:

  • Critical Analysis: They dissect the philosophical underpinnings of perspectivism, drawing parallels to philosophical concepts of truth and subjectivity.
  • Industrial Analogy: They frame data annotation through the lens of "scientific management" (Taylorism), analyzing the division of labor, optimization, and rationalization inherent in large-scale annotation campaigns. This highlights the potential for annotator alienation and exploitation.
  • Ethical and Methodological Critique: They examine the ethical implications of perspectivism, particularly concerning the treatment of annotators and the definition of "truth" in annotated data. They also critique its methodological consequences, arguing it can lead to a confusion between sincerity and truth.
  • Humanities-Inspired Proposals: They advocate for a shift towards a "philological" approach to data, emphasizing corpus construction over mere dataset annotation, and promoting computer-aided corpus analysis.

Datasets

Not discussed in the paper. The paper focuses on the process of annotation and corpus construction rather than specific datasets used for training or evaluation.

Key Insights and Ideas

  • Annotation as Taylorism: Data annotation in NLP, especially at scale, mirrors industrial scientific management, with a division of labor that can lead to annotator alienation and de-skilling.
  • Perspectivism's Ethical Blind Spot: While aiming for inclusivity, perspectivism might inadvertently devalue annotator labor and create a "managerial" approach to ethics rather than addressing fundamental work conditions.
  • Sincerity vs. Truth: The paper argues that perspectivism often conflates the annotator's sincerity (their genuine belief or feeling) with objective "truth," which is a significant epistemological confusion.
  • Corpus Construction Over Annotation: The authors propose shifting focus from annotating pre-existing datasets to actively "constructing" corpora. This involves a more rigorous, intellectual process of defining, selecting, and characterizing texts, akin to philological work.
  • Computer-Aided Corpus Analysis: Leveraging tools from textometrics and digital humanities can create an objective distance, protect annotators from harmful content, and reclassify annotation as intellectual, scientific work.

Key Results

Not discussed in the paper. This is a conceptual and critical paper, not an empirical study reporting quantitative results.

Limitations and Future Work

  • Limitations: The authors acknowledge that their critique is a "short position paper" aiming to "sketch a discussion." They note that the perspectivist literature "only partially solves the ethical problems it raises" and "rather than fighting discrimination, it makes it visible, measurable and computable." They also point out that the "real ethical problem with NLP is the management of work."
  • Future Work: The paper implicitly suggests future work in:
  • Developing more robust methods for "scientific corpus analysis" that revalue interpretative work.
  • Exploring how humanities-inspired methods can be integrated into NLP workflows.
  • Further investigating the ethical implications of data annotation and the management of annotator labor.
  • Shifting the paradigm from "annotating data" to "constructing corpora" as a more scientifically and ethically sound approach.

Summary updated

BibTeX
@inproceedings{valette2024what,
  author = {Mathieu Valette},
  title = {{What Does Perspectivism Mean? An Ethical and Methodological Countercriticism}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure