In standard deep learning, we typically represent categories using one-hot vectors. While intuitive, this approach is geometrically redundant and often arbitrary. If you have three classes, a one-hot encoding places them at the corners of a 3D cube, but the actual decision-making happens on a 2D plane where the sum of probabilities equals one. This mismatch creates a "sum-to-zero" constraint that the network must laboriously learn to satisfy.
A growing body of research suggests a more elegant alternative: mapping classes to the vertices of a regular simplex. By arranging class prototypes as equidistant points in an $(M-1)$-dimensional space, researchers are finding they can improve robustness, speed up convergence, and provide a theoretically grounded "anchor" for feature learning.
The Geometry of Fairness
The core intuition behind simplex-based classification is symmetry. In a binary classification task, we use a single scalar where positive and negative values represent the two classes. They are perfectly balanced across the origin. As we move to $M$ classes, the regular simplex—the generalization of an equilateral triangle (2D) or a regular tetrahedron (3D)—is the unique shape where every vertex is exactly the same distance from every other vertex.
As Simon I. Hill demonstrates in A Framework for Kernel-Based Multi-Category Classification, this geometric arrangement ensures that no class is biased toward another during optimization. Because the targets are equidistant, the decision boundaries are determined solely by the relative distances in feature space, making the entire system rotationally invariant. This isn't just a neat trick; Tomaso Poggio and colleagues show in Multiclass Learning with Simplex Coding that this encoding implicitly satisfies the sum-to-zero constraint, removing the need for explicit optimization penalties or Lagrange multipliers.
Fixed Anchors and Neural Collapse
Traditionally, the final layer of a neural network (the classifier weights) is learned alongside the features. However, recent observations of a phenomenon called Neural Collapse suggest that at the end of training, the last-layer features of deep networks naturally tend to cluster around the vertices of a regular simplex anyway.
If the network is going to end up there, why not start there? Federico Pernici and his team introduced RePoNet in Fix Your Features, arguing that we should fix the final classifier weights as the vertices of a regular polytope from the start. By "fixing the features" to these maximally discriminative positions, the network is forced to learn stationary embeddings that align with an optimal geometric structure. This prevents "feature crowding," where different classes are squeezed into the same corner of the representational space.
Similarly, Xiong Zhou and others proposed Prototype-Anchored Learning (PAL) in icml 2022. By using fixed, geometrically optimal anchors, they provide a "theoretical shield" against noisy labels and class imbalance. Because the anchors are fixed and equidistant, the model cannot easily "collapse" a minority class into the territory of a majority class.
Robustness through Uniform Margins
One of the most compelling reasons to adopt simplex geometry is adversarial robustness. Standard Softmax loss often results in complex, non-linear decision boundaries. These boundaries can have "weak spots"—narrow gaps between classes that an attacker can exploit with tiny perturbations.
In Beyond cross-entropy, Arslan Ali and Tiziano Bianchi introduce the Gaussian class-conditional simplex (GCCS) loss. By forcing features into Gaussian distributions centered on simplex vertices, they create simple, hyperplane decision boundaries. Because every class is equidistant from all others, the "margin" (the distance to the nearest boundary) is maximized and uniform across the entire feature space. This geometric regularity makes it significantly harder for adversarial noise to push a sample across a boundary.
This concept of "maximal margin" is also explored by Tianyu Pang in Max-Mahalanobis Linear Discriminant Analysis Networks. By arranging latent feature means as vertices of a regular simplex, the model achieves a state of maximum inter-class separability, which naturally resists both adversarial attacks and the biases inherent in long-tailed datasets.
Efficiency and Scalability
Beyond theoretical elegance, simplex coding offers practical computational advantages. In Adapting two-class support vector classification methods to many class problems, Simon I. Hill demonstrates that this framework allows researchers to "jump-start" complex global optimizations using the results of fast pairwise classifications.
Furthermore, by reducing the dimensionality of the output space from $M$ to $M-1$, simplex coding removes redundant degrees of freedom. As noted in Maximum Class Separation as Inductive Bias in One Matrix, this can be implemented as a simple, fixed matrix multiplication before the final layer, providing a "maximum separation" inductive bias with zero training overhead. Whether it's for quantum classifiers living on the edge or self-supervised video learning, the regular simplex provides a universal blueprint for organizing high-dimensional information.