TL;DR
This study pits over 40 Large Language Model (LLM) configurations against 73 human annotators to see who better disambiguates the various uses of the pronoun they. While the best LLM (Kimi K3) outperformed the human majority vote and nearly all individual humans, the study concludes that humans are still vital for identifying flaws in research frameworks that models blindly follow.
What Problem Does This Paper Address?
Linguistics research often relies on "annotation"—the manual labeling of text by research assistants to identify patterns. One of the trickiest targets is the English pronoun they. Its usage is currently in a state of flux, shifting between plural (a group), generic singular ("anybody can bring their dog"), and specific singular (referring to a specific non-binary individual or someone whose gender is unknown).
Traditional Coreference Resolution (CRR) systems struggle with these nuances, especially innovative uses. As LLMs become more sophisticated, researchers are asking: Can we finally fire the student assistants and let the models do the heavy lifting? This paper investigates whether LLMs are reliable enough for high-stakes linguistic annotation and where they might still fall short of human intuition.
Methods
The author conducted a controlled experiment comparing human and machine performance on a number-disambiguation task.
- The Task: Participants had to categorize 174 authentic Reddit comments containing a form of they into one of three buckets: Plural, Generic Singular, or Specific Singular.
- The Humans: 73 participants, including local linguistics students (the "gold standard" for student assistants) and Prolific crowdworkers (both self-identified linguists and non-experts).
- The LLMs: Over 40 configurations including the Claude, GPT, Gemini, and Llama families, as well as open-source models like Kimi and DeepSeek.
- Prompting Strategy: The study used a Question-Answering (QA) template. Some models were "context-agnostic," while others were given "Web Search" tools to look up the original Reddit threads for more information.
Datasets
The study utilized a custom dataset of 174 Reddit comments.
- Why Reddit? It provides naturalistic, contemporary language that includes the "innovative" uses of they often missing from older, formal datasets like OntoNotes.
- Balance: The items were curated to ensure a balance between the three categories, though the author notes that in the "wild," plural uses are far more common.
- Ground Truth: Labels were established by the author and a linguistics colleague, achieving a high inter-annotator agreement (Cohen’s kappa of 0.83).
Key Insights and Ideas
- The "Instruction-Following" Trap: LLMs are remarkably good at following instructions—even when those instructions are logically inconsistent. Humans, by contrast, often "underperform" quantitatively because they notice contradictions in the labeling rules and try to infer a more logical path. This makes humans better at methodological debugging.
- Context is a Double-Edged Sword: Surprisingly, giving LLMs access to the full Reddit context (via web search) sometimes lowered their accuracy. The models occasionally "lost the thread" or got distracted by irrelevant information in the discussion, reinforcing the idea that context management is a major bottleneck for LLM reasoning.
- Demographic Nuance: The study specifically recruited young women (ages 20–27) for the human group, as sociolinguistic research identifies them as the most innovative language users and thus the most likely to excel at identifying shifting pronoun uses.
Key Results
- Top Performance: The best LLM, Kimi K3, achieved 94.5% accuracy. This beat the human majority vote (86.2%) and the mean of the top-performing humans (78.5%).
- The Human Champion: Only one human (a local linguistics student) managed to beat the best LLM, scoring 95.7%.
- Efficiency: LLMs were orders of magnitude faster and cheaper. While crowdworkers cost roughly $93 per 100 usable examples (after filtering for inattentiveness), LLMs cost pennies and worked in seconds.
- Innovative Use: While LLMs showed a slight bias toward "standard" plural language, they were not significantly worse than humans at identifying rare and innovative uses of they.
Limitations and Future Work
The author acknowledges several constraints:
- Internal Validity: The analysis relied on aggregated accuracies rather than item-level hierarchical modeling, which might mask specific item difficulties.
- Ambiguity: The dataset excluded "perfectly ambiguous" cases. Future work should investigate how LLMs and humans differ when a sentence truly could mean two different things.
- Human-AI Collaboration: The study suggests that the future isn't "replacement" but "complementarity." Future research should test Human-in-the-Loop (HITL) workflows where LLMs handle the bulk of easy cases, and humans focus on the edge cases that define the boundaries of linguistic theory.
The paper concludes with a warning: if we automate all entry-level annotation, we may lose the training ground where the next generation of expert linguists develops their foundational skills.
Summary updated