Representing Discrete Features in High Dimensional Binary Spaces

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 153 papers58 conceptsfrozen 2026-10-04

Most machine learning algorithms, from neural networks to linear regressors, are fundamentally engines of continuous mathematics. They expect inputs that exist on a spectrum where "2" is twice as much as "1" and halfway to "3." However, the real world is often categorical: a "kitchen" is not mathematically greater than a "bedroom," and a "TCP protocol" is not the average of "UDP" and "ICMP."

To bridge this gap, researchers have developed a suite of techniques to map discrete, qualitative information into high-dimensional binary spaces. This process, often referred to as one-hot or unary encoding, allows models to process categorical data without falling into the trap of "false ordinality"—the incorrect assumption that distinct categories possess an inherent numerical order.

The Problem of False Ordinality

When we represent categories as simple integers (e.g., Red=1, Green=2, Blue=3), we inadvertently tell the model that Blue is "further" from Red than Green is. As Jufeng Yang and colleagues point out in Learning Visual Sentiment Distributions via Augmented Conditional Probability Neural Network, this forces unrelated classes onto a shared numerical scale, which can cripple a model's ability to learn independent weights for each category.

The solution is to expand the feature space. Instead of one dimension with many values, we create many dimensions with only two values (0 or 1). In this high-dimensional space, every category becomes its own axis. This ensures that all categories are equidistant from one another, a concept Viktor Eisenstadt demonstrates is crucial for classifying complex architectural floor plans in Exploring optimal ways to represent topological and spatial features of building designs.

From Tabular Data to Neural Intuition

While decision trees naturally handle categorical splits, neural networks historically struggled with the "messy" heterogeneous data found in spreadsheets. Recent work has focused on "homogenizing" these inputs. Klaus Broelemann and Vadim Borisov introduced DeepTLF in DeepTLF: robust deep neural networks for heterogeneous tabular data, which uses a "TreeDrivenEncoder" to distill the structural logic of boosted trees into binary vectors. This allows standard multi-layer perceptrons to capture non-linear feature interactions that were previously the sole domain of gradient boosting engines.

This representational shift is not just a computational convenience; it mirrors human reasoning. In Human and Machine 'Quick Modeling', Jakob Bernasconi and Karl Gustafson found that human intuition aligns more closely with simple neural networks using unary encoding than with "optimal" logic-based decision trees. By treating attributes as discrete binary units, connectionist models can mimic the way humans categorize the world.

Capturing Non-Additive Complexity

In fields like genetics and chemistry, binary representation does more than just avoid bias—it enables the discovery of complex interactions. Arnór I. Sigurdsson and Ole Lund show in Improved prediction of blood biomarkers using deep learning that one-hot encoding genetic variants allows models to learn "non-additive" effects like dominance and epistasis (gene-gene interactions). If genotypes were represented as continuous numbers, the model would be forced to assume that having two copies of a gene is exactly twice the effect of having one, which is often biologically false.

Similarly, in chemical engineering, Edward Wang and colleagues used one-hot encoding to represent feedstock and catalyst types in Predicting xylose yield from prehydrolysis of hardwoods. This allowed a single neural network to generalize across different chemical species without needing to manually calculate specific kinetic constants for every new material.

Scaling to Sets and Sequences

When dealing with collections of items—such as the set of medications a patient is taking or the tokens in a SQL query—researchers use multi-hot encoding. This expands the binary vector so that multiple positions can be "1" simultaneously.

The Trade-off: Dimensionality vs. Information

The primary challenge of this approach is the "curse of dimensionality." As the number of unique categories (cardinality) grows, the input vector becomes increasingly sparse. Janek Thomas and Bernd Bischl suggest in Automatic Gradient Boosting that automated systems should switch strategies based on cardinality: using simple dummy encoding for few levels, but moving to "impact encoding" (which uses target statistics) when the number of categories becomes unmanageable.

Despite the memory overhead, the transition from scalar labels to high-dimensional binary vectors remains the standard for modern machine learning. By decoupling discrete identities from a shared numerical scale, these representations allow models to treat the world's qualitative diversity with the mathematical precision of continuous space.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_representing_discrete_features_in_high_d_c28bd74e,
  title = {Representing Discrete Features in High Dimensional Binary Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representing-discrete-features-in-high-dimensional-binary-spaces/rdr_c28bd74e06094d3286ba2b47837e06d3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-18},
}

This exact version (version 1, frozen text):

@misc{lacuna_representing_discrete_features_in_high_d_c28bd74e_v1,
  title = {Representing Discrete Features in High Dimensional Binary Spaces},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representing-discrete-features-in-high-dimensional-binary-spaces/rdr_c28bd74e06094d3286ba2b47837e06d3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-18},
}