When we train a machine learning model to classify data into one of $M$ distinct categories, our default instinct is almost always to use one-hot encoding. In a three-class problem, we map the classes to the vectors $(1, 0, 0)$, $(0, 1, 0)$, and $(0, 0, 1)$.
While intuitive, one-hot encoding introduces a subtle geometric redundancy. These three vectors live in a three-dimensional space, yet they all lie flat on a two-dimensional affine plane defined by the equation $x + y + z = 1$. This means one-hot encoding forces the model to predict $M$ values when only $M-1$ degrees of freedom are strictly necessary. To prevent the model's outputs from drifting infinitely along this redundant dimension, optimization frameworks typically must enforce a strict "sum-to-zero" constraint on the model's logits.
Simplex coding solves this problem by matching the dimensionality of the target space to the true degrees of freedom of the classification task. Instead of embedding $M$ classes into an $M$-dimensional space, it maps them to the vertices of a regular simplex in an $(M-1)$-dimensional space.
The Geometry of Fairness
To understand why this geometric shift works, consider how we scale binary classification. In a two-class problem, we map targets to $-1$ and $+1$. Geometrically, this is a 1D line segment—the simplest possible simplex. The decision boundary is a single point at the origin, exactly halfway between both targets.
When we move to three classes, a regular simplex becomes an equilateral triangle embedded in a 2D plane. For $M$ classes, it forms an $(M-1)$-dimensional regular polytope with $M$ vertices.
2 Classes (1D): [ -1 ] ------------ 0 ------------ [ +1 ]
Decision
Boundary
3 Classes (2D): [ Class 1 ]
/ \
/ 0 \ <-- Decision Center
/ \
[ Class 2 ] ------- [ Class 3 ]
As Hill demonstrated in early foundational work (2005, 2007), this regular simplex structure guarantees that every single class target is exactly the same distance away from every other class target. This geometric equidistance ensures absolute fairness during training: no single class is structurally biased toward or away from any other. Furthermore, the optimization landscape becomes completely rotationally invariant, meaning the relative distances between class targets are the only factors dictating how decision boundaries are drawn.
Eliminating Constraints Through Dimensionality
In traditional multiclass setups, managing the redundancy of $M$-dimensional outputs requires explicit mathematical overhead. If a model can add a constant value to all of its class predictions without changing the final classification decision, the optimization problem becomes unconstrained and unstable.
By shifting to an $(M-1)$-dimensional simplex, this issue disappears. As Yang et al. showed, mapping categories directly to the vertices of a regular simplex allows the sum-to-zero constraint to be satisfied implicitly through the geometry itself. Because there are no redundant dimensions, there is no need for explicit Lagrange multipliers or complex constrained optimization routines during training.
This reduction in complexity allows for significant algorithmic efficiency. For instance, Zhu et al. leveraged this geometric simplification to decouple a complex multiclass optimization problem into independent, parallel regression subproblems, drastically speeding up computation for high-dimensional datasets.
Symmetrical Margins and Stability
In binary classification, maximizing the "margin" means pushing data points as far as possible toward either $-1$ or $+1$ and away from the decision boundary. Simplex coding provides a natural, symmetric framework to extend this margin intuition to multiclass settings.
Instead of measuring margins along independent coordinate axes, classification becomes a geometric problem of angular or Euclidean distance relative to fixed points in the $(M-1)$-dimensional space, as noted by Zhang et al.. Work by Saberian et al. and Mroueh et al. demonstrated that minimizing loss functions defined over a regular simplex enforces large margins simultaneously and symmetrically across all categories. This geometric configuration directly correlates with improved robustness against noise and better generalization on unseen data.
This structural symmetry also yields powerful theoretical guarantees. For example, Vigogna et al. proved that when multiclass data features a clear separation between categories, simplex-based encoding allows the classification error to drop to zero exponentially fast as more data is added. Crucially, this convergence occurs without the traditional bias-variance trade-off, offering an explanation for why highly overparameterized models can generalize so effectively.
Versatility Across Domains
Because simplex coding fundamentally cleans up the representation space of multiclass problems, it serves as a robust building block across a wide variety of machine learning paradigms:
- Deep Learning: Hu et al. showed that mapping categorical labels to equidistant points on a high-dimensional hypersphere encourages overparameterized neural networks to learn highly stable, symmetric geometric structures in their feature layers.
- Weakly Supervised Learning: In scenarios where only a subset of classes is labeled, Xu et al. utilized regular simplex spaces to maintain stable, simultaneous margin maximization in multi-positive and unlabeled (MPU) learning.
- Cost-Sensitive Scenarios: When misclassifying certain categories carries higher penalties than others, the geometric flexibility of the simplex allows researchers to distort or weight the target space to account for varying risks without sacrificing optimization stability, as explored by Yang et al..
- Surrogate Loss Analysis: Nueve et al. used the framework of lower-dimensional geometric polytopes to analyze the trade-offs between dimensionality and theoretical consistency in surrogate loss functions.
By replacing the arbitrary coordinate system of one-hot vectors with the intrinsic, equidistant geometry of the simplex, simplex coding aligns the mathematical structure of a classification problem with its true underlying dimensionality.
Key Foundational Work
A Framework for Kernel-Based Multi-Category Classification (2007) — contributed 2 concept elements to this direction.
Adapting two-class support vector classification methods to many class problems (2005) — contributed 2 concept elements to this direction.