Introduction
Ellipsis is a pervasive linguistic phenomenon where speakers omit words that are easily inferable from the context. For example, in the sentence "The engineer completed the project, but the student didn’t," the verb phrase "complete the project" is understood despite being absent in the second clause. While humans resolve these gaps effortlessly using structural cues and real-world knowledge (thematic fit), language models (LMs) face a significant challenge. Ellipsis disrupts the standard compositional mapping between words and meaning, requiring models to perform a "semantic identity procedure" to link the elliptical site back to its antecedent.
This paper introduces ELLie (Ellipsis and its interaction with thematic fit), a novel dataset designed to probe how LMs like BERT and GPT-2 handle ellipsis and whether they leverage thematic typicality to resolve missing information.
The ELLie Dataset
ELLie is the first dataset dedicated to elliptical utterances and their interaction with thematic fit. It is built upon DTFit, a benchmark for thematic fit where human raters evaluated the typicality of verb-argument pairs.
The dataset consists of 575 sentences organized into "quintuplets." Each quintuplet varies the typicality of the event across five conditions: Typical-Typical (T-T), Typical-Atypical (T-AT), Atypical-Typical (AT-T), Atypical-Atypical (AT-AT), and Selectional Preference violation (T-SP_v). ELLie covers various elliptical structures, including VP-ellipsis, Gapping, Sluicing, and Pseudo-gapping, across five thematic roles (Agent, Patient, Instrument, Location, Time).
Methodology
The researchers evaluated GPT-2 (Large) and BERT (Base-cased) using three primary tasks:
- Sentence Typicality Scoring: Assessing the overall probability assigned by the model to sentences across the five typicality conditions.
- Fillers Typicality Scoring: Analyzing the probability of specific "filler" words at both the antecedent and elliptical sites.
- Elided Verb Retrieval: A direct test where models must explicitly predict the missing verb in a cloze-style prompt.
Results and Analysis
Sensitivity to Typicality
Both models demonstrated a clear sensitivity to "event prototypicality." As shown in Figure 6, there is a consistent drop in the probability scores assigned to sentences as they move from typical (T-T) to atypical (AT-AT) and impossible (T-SP_v) conditions. This indicates that LMs can perform a "vibe check," recognizing when a sentence describes an unusual or impossible scenario.
The Reconstruction Gap
Despite their sensitivity to typicality, the models struggled with actual reconstruction. In the Elided Verb Retrieval task, GPT-2 achieved only ~20% accuracy. BERT performed better (~56%), but this was largely due to the prompt providing the direct object as a hint. Even then, BERT failed to include the correct verb in its top five predictions over 32% of the time.
Crucially, when models failed, they often "hallucinated plausibility." For instance, if a prompt involved a butcher and a soldier using a knife, a model might guess the soldier was "cutting meat"—a typical action for butchers/knives—rather than "using the knife," which was the actual elided verb. This suggests models rely on surface-level statistical associations between words rather than a deep structural understanding of the ellipsis.
The Patient Priority
The study found that the "Patient" role (the entity being acted upon) is the most sensitive to atypicality. Models were significantly more likely to flag a strange Patient than a strange Location or Instrument, reflecting the central role of the Patient in defining event typicality.
Conclusion
The findings suggest that while modern LMs are adept at identifying atypical word combinations, they lack the robust structural mechanisms required to resolve ellipsis accurately. They often substitute structural copying with statistical guessing based on thematic typicality. Future work should expand ELLie to include larger-scale human validation and test more advanced Large Language Models (LLMs) to see if increased scale mitigates these structural blind spots.
Summary updated