Abstract
Many crowdsourced NLP datasets contain systematic gaps and biases that are identified only after data collection is complete. Identifying these issues from early data samples during crowdsourcing should make mitigation more efficient, especially when done iteratively. We take natural language inference (NLI) as a test case and ask whether it is beneficial to put a linguist 'in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task. We directly compare three data collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the task, and (iii) an extension of linguist-in-the-loop that provides direct interaction between linguists and crowdworkers via a chatroom. The datasets collected with linguist involvement are more reliably challenging than baseline, without loss of quality. However, we see no evidence that using this data in training leads to better out-of-domain model performance, and the addition of a chat platform has no measurable effect on the resulting dataset. We suggest integrating expert analysis during data collection so that the expert can dynamically address gaps and biases in the dataset.
Introduction
Large-scale Natural Language Understanding (NLU) datasets are typically constructed using non-expert crowdworkers. While cost-effective, this approach often inadvertently introduces systematic biases and shallow heuristics. For instance, in NLI tasks, models frequently learn to predict labels based solely on the hypothesis (ignoring the premise) or by relying on simple lexical overlap. Traditionally, these artifacts are discovered post-hoc, leading to a reactive cycle of adversarial filtering. This research investigates a proactive alternative: integrating expert linguists directly into the data collection loop to identify and mitigate these biases in real-time.
Experimental Design
The researchers conducted a head-to-head comparison of three distinct data collection protocols over five iterative rounds, using RoBERTa models to track progress.
- Baseline: Standard crowdsourcing without expert intervention, mirroring traditional large-scale dataset creation.
- Linguist-in-the-Loop (LitL): Linguists analyzed data after each round using diagnostic models. They then introduced dynamic constraints for subsequent rounds, such as "banned words" to combat hypothesis-only bias and optional "challenge options" (e.g., requiring relative clauses).
- LitL Chat: This protocol extended LitL by adding a Slack chatroom, allowing crowdworkers to interact directly with expert linguists for real-time guidance.
The study utilized the SLATE subset of MNLI for premise sentences, collecting 17.5k examples per protocol. Evaluation was conducted using in-domain validation, diagnostic sets (GLUE, HANS), and out-of-domain sets (MNLI-mismatched, ANLI).
Results and Analysis
Dataset Difficulty and Bias Reduction
A primary finding is that expert-guided protocols successfully produced more challenging data. As shown in Figure 3, models trained and evaluated on LitL and LitL Chat data exhibited lower in-domain accuracy compared to the Baseline. In this context, lower accuracy is a success metric, indicating that the expert-introduced constraints successfully stripped away the "easy" heuristics that models typically exploit.
Furthermore, the LitL protocols significantly reduced hypothesis-only bias. When models were restricted to seeing only the hypothesis, accuracy dropped significantly for the LitL datasets compared to the Baseline. This demonstrates that the dynamic constraints (like banning high-PMI words) forced workers to create more robust examples where the relationship between premise and hypothesis was essential for the label.
Generalization and Cost
Despite the improvements in in-domain difficulty and specific heuristic mitigation (such as improved performance on HANS lexical overlap categories), the interventions did not translate to better out-of-domain generalization. Models trained on LitL data showed no significant performance gains on MNLI-mismatched or ANLI compared to the Baseline.
Additionally, the study found that direct chat communication (LitL Chat) offered a poor return on investment. While workers appreciated the interaction, it nearly doubled the required expert time without yielding measurable improvements in dataset quality over the standard LitL protocol. The cost per example increased by 31.2% for LitL and 58.5% for LitL Chat relative to the Baseline.
Discussion and Conclusion
While putting a linguist in the loop effectively addresses targeted biases and increases dataset difficulty, it is not a silver bullet for model generalization. The low uptake of optional "challenge" constraints (only 21-28%) suggests that financial incentives must be substantial—likely at least 15% of base pay—to motivate workers to tackle complex linguistic tasks. Future work should focus on identifying expert interventions that foster broad reasoning capabilities rather than just patching specific known heuristics. Nonetheless, the study demonstrates that dynamic, expert-led intervention is a viable and effective method for improving the structural integrity of NLU datasets during the collection process.
Summary updated