Introduction
Deep Learning models for Natural Language Processing (NLP) frequently encounter the "curse of dimensionality." Because human languages possess massive vocabularies, input layers become high-dimensional and sparse, making them difficult to train—particularly on small or medium-sized datasets. This sparsity also increases model sensitivity to Out-of-Vocabulary (OOV) words.
Traditionally, researchers have addressed this by clustering similar words (e.g., "run" and "jog") into single tokens. However, this paper challenges that intuition, arguing that grouping similar words creates ambiguity that the model cannot resolve. Instead, the authors propose Diverse Grouping: mapping semantically distinct words into the same "codeword" to reduce dimensionality while preserving the unique meaning of sentences through context.
The Pitfall of Similarity-Based Clustering
In traditional word clustering, words with similar meanings or grammatical roles are grouped together. For example, if "you," "she," and "me" are all assigned to Group A, and "ask" and "tell" are assigned to Group B, the distinct sentences "you ask me" and "she tells me" both collapse into the identical representation "A B A."
As shown in Figure 2, the authors suggest that by grouping diverse words—such as a noun ("cat"), an adjective ("fast"), and a verb ("chase")—the model can use the surrounding context to disambiguate the intended meaning. If "apple" and "run" share a group, the model easily identifies the correct word based on whether the context involves food or movement.
Methods: Algorithms for Diversity
The authors introduce five algorithms across three categories to achieve this mapping:
- Random Grouping: A baseline mapping words to groups randomly, occasionally guided by phonetic encodings like Metaphone.
- Distance-Based Diverse Grouping: Utilizing word embeddings (e.g., word2vec), this method employs a $k$-center clustering approach to find words geometrically furthest from each other in vector space, placing them in the same group.
- Maximum Entropy-Based Grouping: The most advanced approach treats grouping as an optimization problem to maximize the unigram entropy of the representation. This ensures groups are used uniformly, maximizing information retention.
Crucially, the authors prove that the entropy-based objective function is submodular. This mathematical property allows for a greedy approximation algorithm that is guaranteed to stay within a provable bound of the theoretical optimum.
Model Integration and Evaluation
The researchers evaluated these methods across Neural Machine Translation (NMT), POS Tagging, and Language Modeling. The grouped tokens were integrated into standard neural architectures, often working alongside Byte-Pair Encoding (BPE).
As illustrated in Figure 6, the input text $x$ is processed both through standard subword tokenization and the diversity grouping function $\gamma(x)$. These representations are then combined before being fed into the encoder or classifier. This dual-path approach allows the model to benefit from the regularization effects of dimensionality reduction without losing the granularity of subword units.
Key Results
The experiments demonstrated that "diverse" is consistently superior to "similar" for dimensionality reduction:
- Machine Translation: On the IWSLT’17 EN-FR task, the entropy-based method improved the baseline by +6.5 BLEU points, a 36.9% relative improvement. Qualitative analysis showed it nearly doubled the number of "high-quality" translations (BLEU > 20%).
- POS Tagging: The method achieved a 3.6% relative reduction in error rate on the Brown Corpus compared to a standard Bi-LSTM baseline.
- Language Modeling: Perplexity (PPL) was reduced by 3.76%, indicating higher model certainty.
Conclusion and Future Work
By maximizing entropy and leveraging context as a disambiguator, diverse grouping acts as an effective form of regularization. While the current study focused on small-to-medium datasets, future research is needed to determine if these benefits scale to Large Language Models (LLMs) trained on trillions of tokens. Additionally, further optimization of the interaction between BPE and diverse grouping remains an open area for exploration.
Summary updated