Grouping Words with Semantic Diversity

Karine Chubarian, Abdul Rafae Khan, Anastasios Sidiropoulos, Jia XuNAACL 2021 · DOI

Introduction

Deep Learning models for Natural Language Processing (NLP) frequently encounter the "curse of dimensionality." Because human languages possess massive vocabularies, input layers become high-dimensional and sparse, making them difficult to train—particularly on small or medium-sized datasets. This sparsity also increases model sensitivity to Out-of-Vocabulary (OOV) words.

Traditionally, researchers have addressed this by clustering similar words (e.g., "run" and "jog") into single tokens. However, this paper challenges that intuition, arguing that grouping similar words creates ambiguity that the model cannot resolve. Instead, the authors propose Diverse Grouping: mapping semantically distinct words into the same "codeword" to reduce dimensionality while preserving the unique meaning of sentences through context.

The Pitfall of Similarity-Based Clustering

In traditional word clustering, words with similar meanings or grammatical roles are grouped together. For example, if "you," "she," and "me" are all assigned to Group A, and "ask" and "tell" are assigned to Group B, the distinct sentences "you ask me" and "she tells me" both collapse into the identical representation "A B A."

Illustration of diverse grouping where semantically distinct words (e.g., 'cat', 'fast', 'chase') are mapped to the same group to preserve sequence uniqueness through context.

As shown in Figure 2, the authors suggest that by grouping diverse words—such as a noun ("cat"), an adjective ("fast"), and a verb ("chase")—the model can use the surrounding context to disambiguate the intended meaning. If "apple" and "run" share a group, the model easily identifies the correct word based on whether the context involves food or movement.

Methods: Algorithms for Diversity

The authors introduce five algorithms across three categories to achieve this mapping:

  1. Random Grouping: A baseline mapping words to groups randomly, occasionally guided by phonetic encodings like Metaphone.
  2. Distance-Based Diverse Grouping: Utilizing word embeddings (e.g., word2vec), this method employs a $k$-center clustering approach to find words geometrically furthest from each other in vector space, placing them in the same group.
  3. Maximum Entropy-Based Grouping: The most advanced approach treats grouping as an optimization problem to maximize the unigram entropy of the representation. This ensures groups are used uniformly, maximizing information retention.

Crucially, the authors prove that the entropy-based objective function is submodular. This mathematical property allows for a greedy approximation algorithm that is guaranteed to stay within a provable bound of the theoretical optimum.

Model Integration and Evaluation

The researchers evaluated these methods across Neural Machine Translation (NMT), POS Tagging, and Language Modeling. The grouped tokens were integrated into standard neural architectures, often working alongside Byte-Pair Encoding (BPE).

Neural architectures for (a) NMT and (b) POS tagging that incorporate grouped token embeddings alongside standard BPE tokens.

As illustrated in Figure 6, the input text $x$ is processed both through standard subword tokenization and the diversity grouping function $\gamma(x)$. These representations are then combined before being fed into the encoder or classifier. This dual-path approach allows the model to benefit from the regularization effects of dimensionality reduction without losing the granularity of subword units.

Key Results

The experiments demonstrated that "diverse" is consistently superior to "similar" for dimensionality reduction:

  • Machine Translation: On the IWSLT’17 EN-FR task, the entropy-based method improved the baseline by +6.5 BLEU points, a 36.9% relative improvement. Qualitative analysis showed it nearly doubled the number of "high-quality" translations (BLEU > 20%).
  • POS Tagging: The method achieved a 3.6% relative reduction in error rate on the Brown Corpus compared to a standard Bi-LSTM baseline.
  • Language Modeling: Perplexity (PPL) was reduced by 3.76%, indicating higher model certainty.

Conclusion and Future Work

By maximizing entropy and leveraging context as a disambiguator, diverse grouping acts as an effective form of regularization. While the current study focused on small-to-medium datasets, future research is needed to determine if these benefits scale to Large Language Models (LLMs) trained on trillions of tokens. Additionally, further optimization of the interaction between BPE and diverse grouping remains an open area for exploration.

Summary updated

BibTeX
@inproceedings{chubarian2021grouping,
  author = {Karine Chubarian and Abdul Rafae Khan and Anastasios Sidiropoulos and Jia Xu},
  title = {{Grouping Words with Semantic Diversity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.257},
}
Figure