Introduction
Large Language Models (LLMs) are often evaluated on their ability to handle standard linguistic tasks, yet they frequently struggle with "edge cases"—rare linguistic structures that do not follow standard lexical rules. One such phenomenon is the Caused-Motion Construction (CMC). In a sentence like "They laughed him off the stage," the normally intransitive verb "laugh" takes an object ("him") and implies physical movement. While humans intuitively grasp this creative semantic shift, LLMs often fail to recognize the underlying motion component. This paper introduces a cost-effective hybrid pipeline to build high-quality datasets for such rare phenomena and evaluates how modern LLMs perform on these complex syntax-semantics interfaces.
The Challenge of Rare Constructions
CMCs present three primary challenges for automated analysis:
- Sparsity: They appear infrequently in standard corpora compared to prototypical motion verbs.
- Productivity: The construction is highly productive; almost any verb can be used creatively in this frame.
- Syntactic Ambiguity: Many non-CMC sentences share the same surface structure (e.g., "I took that into account"), making simple keyword or pattern matching insufficient.
A Hybrid Annotation Pipeline
To address the difficulty of finding rare examples, the authors developed a multi-stage annotation pipeline designed to maximize the yield of positive examples while minimizing human effort and API costs.
- Dependency-based Filtering: Using spaCy, the authors parsed a massive Reddit corpus to identify a specific syntactic subtree: a verb with a direct object and a prepositional phrase.
- LLM Filtering: Candidate sentences were passed to GPT-3.5. The authors mathematically optimized the prompt to find a "sweet spot" where the combined cost of the API and subsequent human verification was minimized.
- Human Verification: Experts reviewed the LLM's filtered output to establish a "gold standard" dataset of 765 verified CMC sentences.
- Automatic Expansion: By identifying specific 4-tuples (verb, object, preposition, prepositional object) that consistently signaled a CMC, the authors automatically extracted over 127,000 additional high-likelihood instances.
Evaluating LLM Semantic Understanding
To test whether LLMs truly understand the "motion" implied by CMCs, the authors designed a "Motion Test." They asked models: "In the sentence [X], is [Object] moving?" They compared responses for creative CMC verbs (e.g., "sneeze") against prototypical motion verbs (e.g., "throw"). A model with a robust understanding should identify motion in both cases.
Key Results
The evaluation revealed a significant performance gap in modern LLMs:
- Top Performer: Mixtral 8x7b (Instruction Tuned) achieved the highest accuracy at 69.75% (Y→Y).
- Surprising Underperformer: GPT-4 trailed significantly at 57.07%, suggesting that scale alone does not solve these nuanced linguistic challenges.
- Instruction Tuning Nuance: For some models like Llama2, instruction tuning actually decreased performance on understanding the premise of the question, highlighting that fine-tuning can sometimes degrade specific reasoning capabilities.
- Fundamental Failures: Smaller models like Llama2 7b often failed to recognize motion even in prototypical sentences (the X→N category), indicating a lack of basic semantic grounding.
Conclusion
The high error rates observed—exceeding 30% even for the best models—suggest that LLMs are still far from human-level mastery of the syntax-semantics interface. However, the hybrid pipeline demonstrated here provides a scalable, cost-effective framework for linguists to build the specialized datasets necessary to bridge this gap. Future work will expand this methodology to other complex constructions, such as ditransitives and resultatives, to further probe the limits of LLM linguistic competence.
Summary updated