Abstract
Transformer-based language models (LMs) have achieved remarkable success in natural language understanding, yet their ability to process implicit meaning remains poorly understood. This study investigates the phenomenon of complement coercion—where sentences like "The student finished the book" imply an unstated action (e.g., reading). By analyzing surprisal estimates across various critical sentence regions in the GPT-2 model family, we find that while LMs exhibit increased processing costs for coerced sentences, this effect often stems from surface-level statistical confounds rather than human-like semantic reconstruction. Our findings advocate for a "measure more, question more" approach to evaluating linguistic depth in AI.
Introduction: The Challenge of Implicit Meaning
In human communication, we frequently omit details that can be easily inferred from context. A classic example is complement coercion, where a verb that typically expects an activity (like "finished") is paired with an entity noun (like "the book"). Humans automatically "coerce" the object into an event, interpreting the sentence as "finished reading the book."
This mental reconstruction comes with a measurable cognitive cost, often observed in psycholinguistic studies as slower reading times. As Large Language Models (LLMs) become increasingly fluent, a critical question arises: do they truly "read between the lines," or are they simply reacting to the statistical rarity of certain word combinations?
Methodology: Surprisal as a Proxy for Effort
To evaluate how LMs handle coercion, this study employs surprisal—the negative log probability of a word given its preceding context. In information theory, high surprisal serves as a proxy for the "processing cost" or effort required by the model to integrate a word into its current context.
The research focuses on the GPT-2 family (ranging from DistilGPT-2 to GPT-2 XL) and measures surprisal at three critical sentence regions:
- The Differing Verb: The point where the sentence types diverge (e.g., "finished" vs. "read").
- The Target Region: The noun phrase that triggers the coercion (e.g., "the book").
- The Post-target Region: The words immediately following the noun, where "spillover" effects might occur.
The study utilizes three diagnostic datasets adapted from human psycholinguistic experiments to isolate the coercion effect from other linguistic variables, such as noun-type preferences or general semantic anomalies.
Key Findings: Where the Effect Emerges
A primary insight of this research is that the "coercion effect" in LMs is not immediate. If one only examines the verb, the story remains incomplete. The increased surprisal associated with coercion typically manifests at the Target Region (the noun phrase), mirroring human eye-tracking data where readers slow down upon realizing the noun does not perfectly align with the verb's semantic expectations.
As shown in the results, across all five GPT-2 models, surprisal was significantly higher ($p < 0.001$) at the target noun for coerced conditions compared to "preferred" (explicitly stated action) conditions. However, follow-up experiments revealed that this "success" might be misleading. When controlled for noun type, the effect largely vanished; models showed higher surprisal for entity nouns regardless of whether true coercion was required, suggesting they rely on a statistical preference for certain word categories rather than semantic logic.
Conclusion and Future Directions
The study demonstrates that while Transformer-based models can mimic human-like processing costs in complex linguistic scenarios, their underlying mechanisms may rely on surface-level heuristics. LMs appear to treat coerced sentences as "anomaly-lite"—soft violations of rules rather than structures to be resolved through reconstruction.
The "Measure More, Question More" framework suggests that evaluating AI requires multi-stage experimental designs. As we move toward even larger models, it remains essential to verify whether increased scale leads to genuine semantic understanding or merely more sophisticated statistical shortcuts. Future work should continue to bridge the gap between psycholinguistics and NLP to ensure our models are truly capturing the nuances of human language.
Summary updated