Pragmatic Annotation of Articles Related to Police Brutality

Tess Feyen, Alda Mari, Paul PortnerACL 2023

TL;DR

This paper introduces a novel pragmatic annotation task designed to analyze how moral predicates (like "good" or "wrong") are used in news articles about police brutality. By developing a context-specific ontology and annotating sentences from diverse news sources, the researchers aim to understand how political orientation influences the language used to discuss these sensitive topics.

What Problem Does This Paper Address?

Linguistic studies have long been interested in moral predicates, often comparing them to predicates of personal taste or aesthetic judgments. However, there's a significant gap in understanding how these moral terms function in context, particularly in real-world discourse where they carry substantial ethical and social weight. The authors highlight that while we intuitively understand that context influences meaning, the precise relationship between moral predicates and their surrounding discourse, especially in politically charged topics like police brutality, remains underexplored. This lack of understanding makes it difficult to analyze potential biases in media reporting. The paper aims to bridge this gap by investigating how the use and interpretation of moral adjectives in news articles about police brutality might vary depending on the political leanings of the publication.

Methods

The core of this research is the development of a pragmatic annotation task. This involves:

  • Corpus Creation: Gathering news articles discussing the George Floyd (US) and Adama Traoré (France) cases, focusing on the period surrounding the protests (May-August 2020). To capture potential political bias, articles were selected from news sources with known diverging political viewpoints (e.g., Jacobin for left-leaning, Breitbart for right-leaning). To ensure a controlled comparison, for each article containing a moral predicate, a similar article without one was also collected.
  • Ontology Development: A hierarchical ontology was created to categorize the contexts surrounding police brutality. This ontology evolved from initial broad categories to a more refined structure (Figure 1) that distinguishes between entities like "State," "Police," "Justice," "Population," and "Family," among others. The focus is on what is being talked about, rather than who is talking.
  • Annotation Task Design: Sentences from the collected articles were annotated based on the developed ontology. Participants were tasked with assigning each sentence to one of the ontology's categories. The annotation process was designed to be rigorous, with each sentence double-annotated by a research team member and a recruited participant. Participants were not informed about the specific focus on moral predicates to avoid influencing their annotations.
  • Software and Data Handling: Qualtrics was used as the survey software. Python scripts were employed to convert articles into a format suitable for Qualtrics, and NLTK was used for sentence segmentation.

Datasets

The study utilizes a corpus of newspaper articles related to the George Floyd and Adama Traoré cases. Specifically:

  • Corpus Size: Approximately 430 articles in total (294 in English and 136 in French).
  • Selection Criteria: Articles published between May 2020 and August 2020, focusing on the immediate aftermath of the George Floyd protests. For Adama Traoré, articles were selected from the period when his case gained prominence following the BLM protests.
  • Source Diversity: Articles were drawn from news sources with varying political orientations to allow for comparative analysis.

Key Insights and Ideas

  • Context is King for Moral Language: The research strongly suggests that the use and interpretation of moral predicates are highly sensitive to their surrounding context. This echoes findings for predicates of personal taste but extends it to the ethically charged domain of police brutality.
  • Political Leanings May Shape Moral Discourse: The preliminary tests indicated that different news outlets might use moral predicates differently. For instance, Breitbart was observed using "right" more in contexts related to the State, while the New York Times used "good" more in similar contexts. Jacobin and the New York Times, however, used "right" when discussing the population. This hints at a potential correlation between a publication's political orientation and its choice of moral language.
  • Ontology Evolution is Crucial: The initial ontology proved too broad, necessitating refinement to better capture the nuances of police brutality discourse. The iterative development of the ontology, moving from broad categories to more specific ones like distinguishing between "Justice" and "Police," is key to achieving precise annotation.
  • Focus on "What" Not "Who": The ontology's design prioritizes identifying the subject matter of a sentence ("what is being talked about") over the speaker's identity or affiliation. This is crucial for objective analysis, preventing annotators from projecting their own biases.

Key Results

The paper is primarily focused on the elaboration of the annotation task and ontology, rather than presenting final results from the annotation itself. The annotation task is still in progress, and inter-annotator agreement measures are not yet available. However, preliminary tests with the ontology and a subset of sentences did reveal interesting patterns in predicate usage across different news sources, as mentioned in the "Key Insights" section. The authors are using this task to lay the groundwork for future analysis of how moral predicates behave in discussions of police brutality and to compare these behaviors cross-linguistically and across different sub-contexts.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study is limited to police brutality and does not explore other contexts where moral predicates are used.
  • Sentiment Analysis Link: The connection between sentiment analysis and moral predicates has not been extensively studied, and this work does not claim to analyze the polarity of moral predicates, focusing instead on their contextual associations.
  • Cultural and Linguistic Differences: Comparing French and English articles presents challenges due to potential differences in language and cultural contexts, making perfect equivalence difficult. The differing public reception of the George Floyd and Adama Traoré cases highlights this.
  • Data Size and Generalization: The current corpus size (430 articles) is considered limited for broad generalizations. Furthermore, focusing on only one case per country might not fully represent the complexities of police brutality discourse.
  • Translational Nuances: Some moral predicates, like "right" and "just," have multiple translations in French ("juste"), and "good" can be translated as "bon" or "bien," which could impact cross-linguistic comparisons.

Future work will involve analyzing the completed annotations to answer the research questions about the behavior of moral predicates in police brutality contexts and their relationship with political orientation. The authors also plan to conduct a semantic analysis of these predicates.

Summary updated

BibTeX
@inproceedings{feyen2023pragmatic,
  author = {Tess Feyen and Alda Mari and Paul Portner},
  title = {{Pragmatic Annotation of Articles Related to Police Brutality}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure