Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

Leonie Weissweiler, Abdullatif Köksal, Hinrich SchützearXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) are often evaluated on their ability to handle standard linguistic tasks, yet they frequently struggle with "edge cases"—rare linguistic structures that do not follow standard lexical rules. One such phenomenon is the Caused-Motion Construction (CMC). In a sentence like "They laughed him off the stage," the normally intransitive verb "laugh" takes an object ("him") and implies physical movement. While humans intuitively grasp this creative semantic shift, LLMs often fail to recognize the underlying motion component. This paper introduces a cost-effective hybrid pipeline to build high-quality datasets for such rare phenomena and evaluates how modern LLMs perform on these complex syntax-semantics interfaces.

The Challenge of Rare Constructions

CMCs present three primary challenges for automated analysis:

  1. Sparsity: They appear infrequently in standard corpora compared to prototypical motion verbs.
  2. Productivity: The construction is highly productive; almost any verb can be used creatively in this frame.
  3. Syntactic Ambiguity: Many non-CMC sentences share the same surface structure (e.g., "I took that into account"), making simple keyword or pattern matching insufficient.

A Hybrid Annotation Pipeline

To address the difficulty of finding rare examples, the authors developed a multi-stage annotation pipeline designed to maximize the yield of positive examples while minimizing human effort and API costs.

The hybrid annotation pipeline: from raw text to a verified dataset.
  1. Dependency-based Filtering: Using spaCy, the authors parsed a massive Reddit corpus to identify a specific syntactic subtree: a verb with a direct object and a prepositional phrase.
  2. LLM Filtering: Candidate sentences were passed to GPT-3.5. The authors mathematically optimized the prompt to find a "sweet spot" where the combined cost of the API and subsequent human verification was minimized.
  3. Human Verification: Experts reviewed the LLM's filtered output to establish a "gold standard" dataset of 765 verified CMC sentences.
  4. Automatic Expansion: By identifying specific 4-tuples (verb, object, preposition, prepositional object) that consistently signaled a CMC, the authors automatically extracted over 127,000 additional high-likelihood instances.

Evaluating LLM Semantic Understanding

To test whether LLMs truly understand the "motion" implied by CMCs, the authors designed a "Motion Test." They asked models: "In the sentence [X], is [Object] moving?" They compared responses for creative CMC verbs (e.g., "sneeze") against prototypical motion verbs (e.g., "throw"). A model with a robust understanding should identify motion in both cases.

Quantitative evaluation of various LLM families on the CMC motion test.

Key Results

The evaluation revealed a significant performance gap in modern LLMs:

  • Top Performer: Mixtral 8x7b (Instruction Tuned) achieved the highest accuracy at 69.75% (Y→Y).
  • Surprising Underperformer: GPT-4 trailed significantly at 57.07%, suggesting that scale alone does not solve these nuanced linguistic challenges.
  • Instruction Tuning Nuance: For some models like Llama2, instruction tuning actually decreased performance on understanding the premise of the question, highlighting that fine-tuning can sometimes degrade specific reasoning capabilities.
  • Fundamental Failures: Smaller models like Llama2 7b often failed to recognize motion even in prototypical sentences (the X→N category), indicating a lack of basic semantic grounding.

Conclusion

The high error rates observed—exceeding 30% even for the best models—suggest that LLMs are still far from human-level mastery of the syntax-semantics interface. However, the hybrid pipeline demonstrated here provides a scalable, cost-effective framework for linguists to build the specialized datasets necessary to bridge this gap. Future work will expand this methodology to other complex constructions, such as ditransitives and resultatives, to further probe the limits of LLM linguistic competence.

Summary updated

Abstract

Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact that ``sneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.
BibTeX
@misc{weissweiler2024hybrid,
  author = {Leonie Weissweiler and Abdullatif Köksal and Hinrich Schütze},
  title = {{Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena}},
  year = {2024},
  eprint = {2403.06965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.06965},
}
Figure