Scaling Machine Learning to High-Cardinality Categorical Features

79 concepts75 papers1 proposal

Real-world tabular data is notoriously messy. While continuous variables map naturally to mathematical spaces, categorical features with thousands or millions of unique levels—such as zip codes, IP addresses, or medical diagnosis codes—present a fundamental challenge. As noted by Niculescu-Mizil et al. and Lo et al., these high-cardinality attributes frequently cause standard machine learning algorithms to degrade or overfit due to extreme sparsity.

To build effective models, researchers must move away from naive representations and instead exploit statistical regularization or continuous geometric spaces.

The Failure of One-Hot Encoding

The default strategy for handling categorical data is one-hot encoding, which converts each unique category into its own binary indicator column. When cardinality is low (e.g., "day of the week"), this works perfectly. However, when a feature contains thousands of unique levels, one-hot encoding triggers a massive dimensionality explosion.

As Pargent et al. explain, this expansion results in massive, sparse datasets that dramatically increase computational overhead and make models highly susceptible to overfitting. In neural networks, this translates to an unsustainable explosion of input neurons and model parameters, rendering architectures computationally inefficient (Carlin et al.). For tree-based models, determining optimal split points becomes computationally prohibitive without a principled way to group similar categories (Wang et al.). Furthermore, forcing categorical data into high-dimensional binary vectors breaks the low-rank assumptions that many efficient matrix completion and regression algorithms rely on (Townsend et al.; Jayaram et al.).

Statistical Regularization and Mixed Effects

To circumvent the curse of dimensionality, an alternative approach is target encoding, which replaces a categorical value with the expected value of the target variable given that category. However, naive target encoding heavily overfits categories that have very few observations in the dataset.

To solve this, Pargent et al. demonstrate that regularized target encoding—specifically using Generalized Linear Mixed Models (GLMMs)—consistently outperforms traditional methods. It works by shrinking noisy, low-frequency category estimates toward the global mean, effectively pooling information across rare categories.

Taking this statistical insight into deep learning, Simchoni et al. and Rosset et al. introduced the Linear Mixed Model Neural Network (LMMNN). Instead of treating high-cardinality categories as fixed parameters via entity embeddings or one-hot vectors, the authors treat them as random effects. By optimizing a negative log-likelihood derived from mixed models, the network naturally accounts for complex correlation structures and prevents overfitting when observations per category are sparse. This approach has been further scaled using modern Monte Carlo sampling by Tschalzev et al. to handle highly clustered, diverse datasets.

Continuous and Latent Geometric Representations

Another powerful paradigm shifts categorical features away from discrete representations entirely, mapping them instead into dense, continuous spaces that capture latent relationships.

For instance, in generative modeling, discrete one-hot vectors cause severe gradient instabilities. To address this, Si et al. proposed CatConverter, a mechanism that maps high-cardinality categorical features onto a continuous 2D unit circle. This unifies categorical and numerical data into a dense space that accelerates diffusion model training and improves synthetic data quality. Similarly, Mueller et al. and Sattarov et al. leverage continuous embedding representations to scale tabular diffusion models without the parameter explosion typical of discrete encodings.

Alternatively, categories can be embedded by analyzing their structural relationships. Bromuri et al. developed Graph-based Categorical Embedding (GCE), which treats feature values as nodes in a graph and uses graph neural networks to learn embeddings that capture multi-hop relationships between categories.

This geometric smoothing is also vital in Bayesian optimization, where high-cardinality discrete spaces typically break down (Garrido-Merchán et al.). Models like the Categorical Latent Gaussian Process (CLGP) developed by Gal et al. map discrete observations into a continuous latent space via non-linear Gaussian Processes, resolving the extreme sparsity problem by allowing the model to generalize across nearby points in the latent space.

Key Foundational Work

An Ensemble of Three Classifiers for KDD Cup 2009: Expanded Linear Model, Heterogeneous Boosting, and Selective Naive Bayes (2009) — contributed 1 concept elements to this direction.
On the use of a clinical kernel in survival analysis (2010) — contributed 1 concept elements to this direction.
Winning the KDD Cup Orange Challenge with Ensemble Selection (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_cardinality_categorical_encoding_b7adaab2,
  title = {high-cardinality categorical encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-cardinality-categorical-encoding/rdr_b7adaab23e504baa9a983862e7031dd4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_cardinality_categorical_encoding_b7adaab2_v1,
  title = {high-cardinality categorical encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-cardinality-categorical-encoding/rdr_b7adaab23e504baa9a983862e7031dd4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake