Tools Impact on the Quality of Annotations for Chat Untangling

Jhonny Cerezo, Felipe Bravo-Marquez, Alexandre Henri BergelACL 2021

TL;DR

This paper investigates how annotation tools affect the quality of data for chat untangling, a task crucial for understanding multi-participant conversations. They introduce a new tool, Parlay, and compare it to an existing one, SLATE, finding that while annotation quality is similar, Parlay offers a significantly better user experience.

What Problem Does This Paper Address?

Creating high-quality annotated datasets for Natural Language Processing (NLP) tasks is notoriously time-consuming and expensive. While the quantity and quality of data are known to limit model performance, less attention has been paid to the role of annotation tools in this process. Specifically, for the task of "chat untangling" – identifying distinct conversations within a multi-participant chat – annotators need to maintain a complex mental model. This paper explores how different annotation tools can either aid or hinder this process for non-expert annotators, impacting both the quality of the annotations and the annotator's experience.

Methods

The core of this study is a controlled experiment comparing two annotation tools:

  • SLATE: An existing command-line based tool where interactions are primarily driven by keyboard commands, and messages are displayed as raw text with color-coding for annotations.
  • Parlay: A newly developed tool that integrates mouse interaction and visual links (arcs) to represent connections between messages.

Twelve non-expert participants were recruited and underwent an introductory session to understand the chat untangling task and annotation guidelines. They then annotated 100 messages using each tool in separate sessions. The evaluation methodology focused on:

  • Usability: Measured using the System Usability Scale (SUS).
  • Task Load: Assessed using the NASA-Task Load Index (NASA/TLX).
  • Annotation Time: Logged automatically.
  • Annotation Performance: Evaluated by comparing annotator outputs to a gold standard dataset using Cohen's Kappa coefficient (κ).

Datasets

The study used an adjudicated dataset from Kummerfeld et al. (2019) for chat untangling. This dataset is considered the gold standard for evaluating inter-annotator agreement. The authors selected pairs of files from this dataset that exhibited high similarity based on annotator agreement with the gold standard and the number of annotated conversations.

Key Insights and Ideas

  • User Experience Matters: While both tools produced comparable annotation quality, Parlay significantly outperformed SLATE in terms of user experience, as indicated by higher usability scores and lower task load.
  • Visual Aids are Beneficial: Participants highlighted Parlay's visual representation of links (blue arcs) and mouse interaction as key factors contributing to its better user experience. This suggests that intuitive visual cues can greatly simplify complex annotation tasks.
  • Experience Trumps Tool (for Performance): The study found that participants' performance (annotation quality) improved significantly with experience gained across the annotation sessions, regardless of the tool used. This implies that for non-expert annotators, the learning curve associated with the task itself is a more dominant factor in performance than the specific tool.
  • No Significant Tool Impact on Performance: Contrary to what might be expected, the annotation tool itself did not show a statistically significant impact on the quality of the annotations. This suggests that for this specific task and user group, the inherent complexity of chat untangling and the annotator's learning curve played a larger role.

Key Results

  • Annotation Quality: No significant difference was found in annotation performance (measured by Cohen's Kappa) between Parlay and SLATE (p=0.1583).
  • Experience Gain: A significant difference was observed in annotation quality between the first and second evaluation sessions (p=0.007636), indicating that annotators improved with experience.
  • Annotation Time: SLATE was slightly faster for annotation (Mean=41.083 min) compared to Parlay (Mean=48.5 min). However, the second session was faster overall for both tools.
  • Usability: Parlay received higher SUS scores, indicating better perceived usability. Participants rated Parlay higher on positive usability aspects and lower on negative ones.
  • Task Load: Parlay generally reported lower task load across most NASA/TLX dimensions, with participants feeling they performed better using Parlay.

Limitations and Future Work

  • Non-Expert Annotators: The study focused on non-expert annotators. The impact of these tools on expert annotators remains an open question.
  • Specific Task: The findings are specific to the chat untangling task. Generalizability to other NLP annotation tasks would require further investigation.
  • Sample Size: While 12 participants is reasonable for a controlled experiment, a larger sample size could provide even more robust statistical power.

For future work, the authors suggest studying the inter-annotator agreement for each tool and investigating the discrepancies between expert and non-expert annotations. They also aim to further explore how annotation tools influence the quality of chat untangling data.

Summary updated

BibTeX
@inproceedings{cerezo2021tools,
  author = {Jhonny Cerezo and Felipe Bravo-Marquez and Alexandre Henri Bergel},
  title = {{Tools Impact on the Quality of Annotations for Chat Untangling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure