TL;DR
This paper investigates whether GPT-4, through Microsoft Copilot, can understand the linguistic phenomenon of Aspectual Coercion, specifically the sentence "John began the book." While GPT-4 demonstrates an impressive ability to generate plausible interpretations and understand contextual nuances, it struggles to provide deeper linguistic theoretical explanations for these phenomena, highlighting the gap between distributional knowledge and true linguistic comprehension.
What Problem Does This Paper Address?
The paper tackles the challenge of Aspectual Coercion, a linguistic phenomenon where a verb expects a certain type of complement (e.g., an "event" type), but is given a complement of a different type (e.g., an "entity" type). For instance, the verb "begin" typically takes an event as its complement, but in the sentence "John began the book," "the book" is an entity. Native speakers understand this as "John began reading the book" or "John began writing the book," inferring the missing event based on linguistic conventions and world knowledge. This inference process, known as Type Coercion, is complex and relies on implicit understanding. The paper questions whether advanced Large Language Models (LLMs) like GPT-4 can grasp these subtle linguistic insights beyond simply recognizing word co-occurrence patterns.
Methods
The researchers used a prompt-based approach, interacting with Microsoft Copilot (which integrates GPT-4) to probe its understanding of Aspectual Coercion. They designed a series of questions and prompts to:
- Examine co-occurrence constraints: Testing GPT-4's knowledge of which verbs can and cannot combine with "begin" and "book."
- Assess interpretation capabilities: Asking for the meaning of "John began the book" in various contexts (default, specific situational, and professional).
- Probe theoretical understanding: Inquiring about the underlying linguistic principles and constraints governing Aspectual Coercion.
- Evaluate context sensitivity: Testing if GPT-4 could adjust its interpretations based on provided context.
Datasets
Not explicitly discussed as training datasets. The study uses the sentence "John began the book" and variations as the primary "data" for interaction with GPT-4.
Key Insights and Ideas
- Distributional vs. Theoretical Understanding: GPT-4 excels at identifying common co-occurring verbs and generating plausible interpretations based on distributional patterns (e.g., "read," "write" with "book"). However, it struggles to articulate the why behind these patterns from a linguistic theory perspective.
- Context is King (for GPT-4): GPT-4 demonstrates a strong ability to adapt its interpretations based on provided context. For example, when told John is an editor, it correctly infers "editing the book" as a likely interpretation.
- Ambiguity Recognition: GPT-4 acknowledges that sentences like "John began the book" are ambiguous and can have multiple valid interpretations, especially without specific context.
- Limitations in Constraint Recognition: GPT-4 appears to have difficulty understanding strict linguistic constraints, such as the fact that "begin" cannot take achievement or punctual event-type complements. It sometimes incorrectly labels sentences as semantically acceptable when they violate these rules.
- Potential for Lexical Resource Building: The study suggests that LLMs could be valuable tools for constructing linguistic resources related to phenomena like Aspectual Coercion, by generating diverse interpretations and identifying potential candidates for elided verbs.
Key Results
- GPT-4, via Copilot, could generate lists of common and uncommon verbs that co-occur with "begin" and "book," demonstrating an understanding of typical usage.
- It provided multiple plausible interpretations for "John began the book," including "reading," "writing," and even more creative ones like "starting a book club."
- When provided with context (e.g., John as an editor), GPT-4 could infer more specific and appropriate interpretations (e.g., "editing the book").
- However, GPT-4 failed to identify that "John began to obtain the book" is semantically problematic from a theoretical standpoint, despite correctly identifying that "John obtained the book for three hours" is not acceptable due to the instantaneous nature of "obtain."
Limitations and Future Work
- Lack of Deep Linguistic Theory: The primary limitation identified is GPT-4's inability to provide sophisticated linguistic theoretical explanations for Aspectual Coercion, relying more on distributional properties.
- Mode Dependency: Copilot's responses can vary slightly depending on its mode (balanced, precise, creative), and it can remember or forget previous interactions, making consistent assessment challenging.
- Future Work: The authors propose expanding research to include other LLMs, integrating linguistic knowledge into LLMs to enhance explainability, and developing more comprehensive prompt-based learning strategies. They also suggest comparing GPT-4 with other models like Chat-GPT.
Summary updated