Annotating picture description task responses for content analysis

Levi King, Markus DickinsonNAACL 2018

TL;DR

This paper introduces the SAILS Corpus, a new dataset of picture description task responses from native and non-native English speakers, annotated with five features to capture variability in linguistic form and meaning. The work highlights the challenges of analyzing learner language and proposes a robust annotation scheme that achieves high inter-annotator agreement, paving the way for better computational analysis of learner writing.

What Problem Does This Paper Address?

Analyzing second language learner data presents a significant challenge due to the inherent variability in how learners express meaning. This variability makes it difficult for computational systems to accurately assess grammatical correctness, linguistic patterns, and content. Specifically, understanding how linguistic form maps to meaning, and how this mapping varies across native speakers (NS) and non-native speakers (NNS), is crucial for developing effective language learning technologies. The authors aim to address this by creating a controlled dataset and a detailed annotation scheme to systematically analyze this variability.

Methods

The core of this work is the development of a detailed annotation scheme for picture description task (PDT) responses. The authors designed five binary features to capture different aspects of response quality:

  • Core Event: Does the response capture the main action depicted in the image?
  • Verifiability: Does the response contain only information that is true and verifiable from the image?
  • Answerhood: Does the response make a clear attempt to answer the question posed?
  • Interpretability: Does the response evoke a clear mental image?
  • Grammaticality: Is the response free from errors of spelling and grammar?

This scheme was developed iteratively with input from annotators and language professionals. The authors then had two native English speakers annotate the collected responses, and they report on inter-annotator agreement to validate the scheme's reliability. They also calculated type-to-token ratios (TTR) to quantify response variation within NS and NNS groups.

Datasets

The paper introduces the SAILS Corpus, a new dataset comprising approximately 16,000 picture description task (PDT) responses from 499 participants.

  • Non-Native Speakers (NNS): 141 participants, primarily native Mandarin Chinese speakers, recruited from intermediate and advanced ESL courses at Indiana University.
  • Native Speakers (NS): 358 participants, with some known to the researchers and the majority recruited via an online survey platform.

The PDT itself consists of 30 cartoon-like vector graphics designed to elicit specific linguistic behaviors, divided into intransitive, transitive, and ditransitive actions. The images were modified to remove non-essential details and text to control for meaning.

Key Insights and Ideas

  • Variability is complex and multi-faceted: The five annotation features highlight that evaluating a response involves more than just grammatical correctness; it encompasses how well the response captures the core event, its verifiability, clarity, and directness in answering the prompt.
  • NS responses show more variation: Despite NNSs often being perceived as more variable, the study found that NSs, when asked for multiple responses, exhibit greater variation in their linguistic output, suggesting that asking for more than one response is an effective way to capture a broader range of native-like expressions.
  • Task constraints matter: Targeted questions (e.g., "What is the boy doing?") constrain response variation more than untargeted questions (e.g., "What is happening?"), as evidenced by lower TTRs for targeted items.
  • Complexity increases variation and decreases agreement: As the complexity of the depicted action increases (intransitive < transitive < ditransitive), so does the variation in responses (higher TTRs) and, interestingly, the difficulty for annotators, leading to lower agreement scores.
  • Disagreements reveal subtle linguistic challenges: The analysis of annotator disagreements provides valuable insights into the nuances of language, such as the fine line between inference and speculation (Verifiability) or the precise definition of a "core event" (Core Event).

Key Results

  • High Inter-Annotator Agreement: The annotation scheme achieved high raw agreement scores (ranging from 91% to 99.3%) and Cohen's kappa scores (above 0.74 for all features), indicating the scheme is reliable and robust.
  • Answerhood has the highest agreement: This feature, focused on whether the response directly addresses the prompt, showed the highest agreement scores (kappa = 0.936), suggesting it's the most straightforward to assess.
  • Interpretability has the lowest agreement: This feature, requiring annotators to judge if a response evokes a clear mental image, had the lowest agreement scores (kappa = 0.744), highlighting its subjective nature.
  • NNS responses are more "correct" in some aspects: Surprisingly, NNS responses received higher "yes" rates for core event, verifiability, and answerhood compared to NS responses. This is attributed to the NNS participants' motivation and the controlled environment, as well as the NSs potentially submitting less-effortful second responses.
  • Grammaticality agreement is higher for NNS: Annotators agreed more on the grammaticality of NNS responses (kappa = 0.863) than NS responses (kappa = 0.786), likely because NNS grammatical errors are often more obvious.

Limitations and Future Work

  • NNS participant demographics: The NNS group was heavily dominated by Mandarin Chinese speakers, which may limit the generalizability of findings to other NNS populations.
  • NS response motivation: The NS participants, particularly those recruited online, may have had different motivations and fatigue levels, potentially affecting the quality and variability of their responses.
  • Ambiguity in feature definitions: While the annotation scheme is robust, some disagreements highlight the inherent subjectivity in defining concepts like "core event" and "interpretability," suggesting that clearer guidelines or more specific examples might further improve agreement.
  • Future work: The authors suggest that the SAILS Corpus and its annotations can be used for various NLP tasks beyond content analysis, including question answering, dialog systems, and pragmatic modeling. They also propose expanding the feature set and the task itself to include more items. For specific applications like language testing, they recommend using expert annotators and constructed gold standards.

Summary updated

BibTeX
@inproceedings{king2018annotating,
  author = {Levi King and Markus Dickinson},
  title = {{Annotating picture description task responses for content analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure