A Study on Annotation Interfaces for Summary Comparison

Sian Gooding, Lucas Werner, Victor CărbuneACL 2023

TL;DR

This paper investigates how user interface design impacts annotator agreement in summary comparison tasks, crucial for training large language models via Reinforcement Learning from Human Feedback (RLHF). They found that while a novel interface with a token highlighting feature was perceived as useful, it surprisingly led to lower annotator agreement, suggesting that task complexity and unfamiliarity play significant roles.

What Problem Does This Paper Address?

Evaluating the quality of automatically generated summaries is notoriously difficult. While human evaluation is the gold standard, achieving consistent agreement among human annotators is a significant challenge. This is particularly problematic for Reinforcement Learning from Human Feedback (RLHF), a technique increasingly used to train large language models (LLMs). RLHF relies on collecting human preference data, often through pairwise comparisons of summaries. However, the effectiveness of this data hinges on the quality and consistency of human judgments. Previous work has shown that even expert annotators can struggle to agree, and the usability of the interface used for these comparisons is often overlooked. This paper aims to explore how the design of the annotation interface influences annotator agreement and engagement in summary comparison tasks.

Methods

The study employed a comparative approach using two interface settings.

  • Baseline Interface (Interface 1): This is a standard interface that annotators had been using for six months prior to the study. It likely involves presenting summaries and asking annotators to select the best one.
  • Novel Interface (Interface 2): This interface was designed in collaboration with annotators to address perceived issues with readability and to facilitate quicker identification of overlaps between summaries and the original text. A key feature of this interface is the ability to highlight tokens (words) in the original text and see their instances highlighted across the generated summaries. This aims to help annotators quickly assess how well a summary captures the original content.

The study involved 6 in-house annotators who were trained and paid for their work. They were tasked with comparing summaries generated by various LLMs and human-written gold standards. The researchers measured annotator engagement (e.g., time spent, use of the highlighting tool) and inter-annotator agreement (using Kappa statistics).

Datasets

The datasets used in this study consisted of two batches, each containing 500 samples. These samples comprised articles scraped from the web, along with summaries generated by fine-tuning language models such as LaMDA (150B), Pegasus (500M), and FlanT5 XXL. The models used were Transformer Encoder models with LSTM Decoders, containing approximately 27 million parameters. The fine-tuning data for these models consisted of 1-10K gold standard summaries.

Key Insights and Ideas

  • Interface Usability vs. Agreement: While annotators found the new highlighting feature useful, its introduction surprisingly led to a decrease in inter-annotator agreement. This suggests that perceived usefulness doesn't always translate directly to improved agreement, and other factors like task complexity are critical.
  • Task Complexity Matters: The novel interface also introduced a more complex task: selecting the top n summaries instead of just the single best one. This increased cognitive load is posited as a significant contributor to the reduced agreement, highlighting the interplay between interface design and task formulation.
  • Familiarity Breeds Agreement: Annotator agreement tended to increase with greater exposure to the new interface, indicating that familiarity and practice are crucial for stabilizing judgments, even with a novel tool.
  • Token Highlighting Behavior: Annotators used the token highlighting tool with varying degrees of engagement. Some annotators heavily utilized it, while others barely touched it. This variability provides insights into individual annotation strategies and potential reasons for agreement differences.
  • Noun Preference in Highlighting: When annotators did use the highlighting tool, they tended to select nouns, suggesting a focus on content words when assessing summary quality.

Key Results

  • Agreement Drop: Inter-annotator agreement, measured by pairwise Kappa, was higher for the baseline interface (average Kappa of 0.59) compared to the novel interface (average Kappa of 0.46).
  • Time Increase: Annotators generally took longer to complete tasks with the novel interface, which is attributed to both the new features and the more complex task of selecting multiple summaries.
  • Token Selection: A total of 1836 tokens were selected across all annotators. The study found a statistically significant correlation between the number of times a token was selected and its frequency in the original article. Nouns constituted 64% of the selected tokens.
  • Agreement and Uncertainty: The study found a statistically significant relationship between annotator agreement and the frequency of changes made during the annotation process, suggesting that behavioral indicators of uncertainty can predict agreement levels.

Limitations and Future Work

The authors acknowledge that this was a pilot study, and drawing definitive conclusions about the primary factor impacting agreement is difficult due to the variations in experimental settings and interfaces. They also note that the observed lack of adoption of the highlighting tool by some annotators might be due to unfamiliarity rather than a lack of utility, which could change over time.

Future work will involve:

  • Conducting longitudinal studies to examine if annotator agreement improves with experience.
  • Investigating how annotator behaviors (e.g., time taken, token selection frequency, summary selection changes) can provide insights into task difficulty and predict agreement.
  • Exploring the use of implicit interaction signals to predict annotator agreement.
  • Further experiments to disentangle the effects of interface design versus task complexity on agreement.

Summary updated

BibTeX
@inproceedings{gooding2023study,
  author = {Sian Gooding and Lucas Werner and Victor Cărbune},
  title = {{A Study on Annotation Interfaces for Summary Comparison}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure