Does Putting a Linguist in the Loop Improve NLU Data Collection?

Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alex Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman

Abstract: Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, Samuel R. Bowman. Findings of the Association for Computational Linguistics: EMNLP 2021.

2021.

Introduction

Natural Language Inference (NLI) datasets are the backbone of modern language model evaluation, yet they are frequently plagued by "statistical shortcuts" or artifacts. These flaws occur when models learn to exploit simple patterns—such as associating the word "not" with a contradiction label—rather than performing genuine linguistic reasoning. While these issues are typically identified after data collection is complete, this paper investigates a proactive approach: involving expert linguists directly in the crowdsourcing loop to steer workers toward more robust and complex data generation.

Methodology

The researchers evaluated three distinct data collection protocols over five iterative rounds, utilizing RoBERTa-Large for model diagnostics and evaluation. The source premises were drawn from the SLATE subset of the Multi-Genre NLI (MNLI) corpus.

  1. Baseline: Standard crowdsourcing without expert intervention.
  2. Linguist-in-the-Loop (LitL): Linguists performed diagnostic error analysis after each round. They then introduced "Banned Words" (statistically biased terms) and "Challenge Options" (linguistic constraints like avoiding word overlap) to guide crowdworkers.
  3. LitL Chat: This protocol mirrored LitL but added a Slack channel for real-time, direct communication between linguists and crowdworkers.
The hierarchical structure of the three experimental protocols: Baseline, LitL, and LitL Chat.

Results and Discussion

Data Difficulty and Artifact Suppression

The LitL interventions successfully produced more challenging datasets. Model accuracy on LitL-validated data was approximately 5 points lower than on the Baseline data, indicating a more rigorous stress test for NLI systems. Crucially, the approach reduced "hypothesis-only" artifacts; in the Baseline, models could often predict labels by looking at the hypothesis alone, whereas this "cheating" accuracy was significantly lower in LitL data.

Model accuracy across rounds for in-domain validation and hypothesis-only inputs, showing the increased difficulty of LitL data.

The Generalization Gap and the Chatroom Paradox

Despite creating higher-quality, "linguistically honest" data, models trained on LitL datasets did not show improved generalization on external benchmarks like ANLI or MNLI-mismatched. This suggests that the specific artifacts addressed by linguists in this study may not be the primary drivers of model failure in out-of-domain scenarios.

Furthermore, the study revealed a "Chatroom Paradox": providing a direct Slack channel for workers (LitL Chat) offered no measurable benefit over providing written constraints (LitL). This is a positive finding for scalability, as well-crafted iterative instructions are far less labor-intensive than one-on-one mentoring.

The Cost of Quality

Expertise comes with a financial trade-off. Accounting for expert labor at $40/hr, the cost per example increased by 31.2% for LitL and 58.5% for LitL Chat compared to the baseline.

Conclusion

While putting a linguist in the loop effectively suppresses known artifacts and creates more challenging evaluation sets, it does not yet provide a "silver bullet" for model generalization. Future research should focus on developing interventions specifically targeted at out-of-domain robustness and exploring how Large Language Models might automate the linguist's role to provide real-time feedback more affordably.

Summary updated

BibTeX
@inproceedings{parrish2021does,
  author = {Alicia Parrish and William Huang and Omar Agha and Soo-Hwan Lee and Nikita Nangia and Alex Warstadt and Karmanya Aggarwal and Emily Allaway and Tal Linzen and Samuel R. Bowman},
  title = {{Does Putting a Linguist in the Loop Improve NLU Data Collection?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=j5xASlViSNS},
}
Figure