Will it Unblend?

Yuval Pinter, Cassandra L. Jacobs, Jacob Eisenstein

Introduction

Modern Natural Language Processing (NLP) systems rely heavily on subword tokenization (e.g., WordPiece or BPE) to handle out-of-vocabulary words. While effective for standard morphology (un-break-able) and simple compounds (watermelon), these methods struggle with blends—portmanteaux like shoptics (shop + optics) or innoventor (innovator + inventor). In a blend, the fusion process often destroys character-level information at the boundary, leaving models "semantically impoverished." This paper investigates whether state-of-the-art models like BERT can recognize the structure of a blend and recover its original base words.

Methods and Datasets

The researchers utilized the NYTWIT dataset, which contains novel words from the New York Times. The dataset includes 142 blends, 121 "transparent" compounds (e.g., quizmaker), and 49 "opaque" compounds (e.g., deathbox).

To analyze these words, the authors introduced a labeling schema called PAXOBS (Prefix, Base A, shared X, Base B, Suffix) to precisely map characters to their respective base words.

Examples of blends, their bases, semantic relations, and PAXOBS segmentation tags.

The study employed three primary experimental lenses:

  1. Representational Analysis: Comparing BERT’s internal representation of a blend to the average representation of its bases.
  2. Segmentation: Testing how well tokenizers can identify the boundaries defined by PAXOBS.
  3. Base Recovery (BERT RANKER): A novel unsupervised method where BERT ranks candidate base word pairs from a dictionary based on sentence context and the blend's spelling.

Key Insights: Form over Function

A central finding of the paper is that BERT’s difficulty with blends is primarily a form-level pathology. To test this, the authors created "Smoothies"—synthetic blends generated by forcing standard compounds to lose characters.

Similarity between raw and decomposed representations for real blends/compounds (left) and synthetic "smoothies" (right) across BERT layers.

As shown in the figure above, when a simple compound is mangled to look like a blend (a "smoothie"), BERT’s performance drops significantly, matching its performance on real blends. This suggests that the semantic novelty of a blend is less problematic for the model than the fact that its orthography (spelling) is distorted.

Key Results

  • Subtokenization Blindness: Standard tokenizers like WordPiece are largely ineffective at segmenting blends, achieving only about 22% exact match accuracy. They are frequently misled by plausible sub-strings that do not represent the actual base words.
  • The Power of Context: For recovering original words, the BERT RANKER significantly outperformed static embeddings. By using sentence context (e.g., identifying eggs + accessories for eggcessories when the sentence mentions breakfast), the ranker achieved a Precision@1 of 26.4%, a massive improvement over the 1.4% baseline.
  • Segmentation Performance: Domain-specific subword models trained on news data performed best for boundary detection, but still only reached a ~56% F1 score, indicating that blend segmentation remains a significant challenge for current architectures.

Conclusion and Future Work

The study concludes that current NLP models are ill-equipped to handle the non-linear character fusions found in portmanteaux. Future improvements may require moving beyond simple subtokenization toward models that incorporate phonetic information—since blends are often formed based on sound—and larger, more diverse datasets of fused linguistic forms.

Summary updated

Abstract

Natural language processing systems often struggle with out-of-vocabulary (OOV) terms, which do not appear in training data. Blends, such as "innoventor", are one particularly challenging class of OOV, as they are formed by fusing together two or more bases that relate to the intended meaning in unpredictable manners and degrees. In this work, we run experiments on a novel dataset of English OOV blends to quantify the difficulty of interpreting the meanings of blends by large-scale contextual language models such as BERT. We first show that BERT's processing of these blends does not fully access the component meanings, leaving their contextual representations semantically impoverished. We find this is mostly due to the loss of characters resulting from blend formation. Then, we assess how easily different models can recognize the structure and recover the origin of blends, and find that context-aware embedding systems outperform character-level and context-free embeddings, although their results are still far from satisfactory.
BibTeX
@inproceedings{pinter2020will,
  author = {Yuval Pinter and Cassandra L. Jacobs and Jacob Eisenstein},
  title = {{Will it Unblend?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.138},
}
Figure