Introduction
Modern Natural Language Processing (NLP) models, particularly Transformers like BERT, excel at capturing contextual nuances. However, they face a significant challenge with out-of-vocabulary (OOV) terms. While standard subtokenization strategies (e.g., WordPiece) effectively decompose regular compounds like watermelon into water + melon, they struggle with blends (portmanteaux).
Blends like hatriotism (hate + patriotism) or shoptics (shop + optics) involve "pathological" character loss or sharing. This research investigates whether large-scale language models can recover the underlying components of these creative formations or if the structural mangling of the word form renders them unintelligible to current subtokenizers.
Methodology
The study employs four primary approaches to analyze how models process blends:
- Representational Analysis: Measuring layer-wise similarity between a blend and its constituent bases across BERT’s 12 layers.
- The "Smoothie" Control: To isolate the impact of character loss from semantic shifts, the authors created "smoothies"—artificial blends generated by forcing character deletions on standard, transparent compounds (e.g., quizmaker).
- PAXOBS Schema: A character-level labeling system designed to track the provenance of each character in a blend: Prefix, Base A, Shared (X), Base B, and Suffix.
- BERT Ranker: An unsupervised recovery method that replaces a blend in a sentence with two
[MASK]tokens, tasking BERT with predicting the most probable base pair.
Datasets
Evaluation was conducted using two primary sources:
- NYTWIT: A curated dataset of 311 neologisms from the New York Times, including 142 blends and 170 compounds (transparent and opaque), annotated with semantic relations.
- Gangal et al. (2017): A larger corpus of approximately 1,500 blends used for training supervised sequence taggers.
Analysis: Form vs. Meaning
A central finding of the paper is that BERT’s difficulty with blends is primarily a matter of form, not semantics. In representational tests, BERT's performance on "smoothies" (mangled compounds) dropped to match its poor performance on real blends. This suggests that the "seams" of the word are structurally vital; when character loss obscures these boundaries, the model loses access to the underlying conceptual representations.
Key Results
- Semantic Impoverishment: Lower layers of BERT show significantly less similarity between a blend and its bases compared to standard compounds. Similarity only begins to converge in the final layers, likely due to contextual signals.
- Segmentation Fragility: Subtokenizers are remarkably brittle when faced with non-linear morphology. WordPiece achieved only a 22% exact match rate in identifying the internal boundaries of blends.
- Contextual Recovery: Context acts as a vital lifeline. In roughly 40% of cases, one of the original base words appears elsewhere in the document. Leveraging this, the BERT Ranker achieved a Precision@1 of 0.264, outperforming static embeddings and edit-distance baselines, though leaving 75% of blends still "unsolved."
Conclusion and Future Work
The study confirms that while subtokenization handles additive morphology well, it is ill-equipped for the creative, subtractive ways humans mash words together. Future research may need to incorporate phonetic information—as many blends are formed based on sound rather than spelling—and expand beyond two-base recovery to handle complex formations like fanimatic (fan + animation + cinematic).
Summary updated