Aligning Feature Spaces with Fixed Geometric Classifiers

47 concepts31 papers1 proposal

In standard deep learning classification, a network learns two things simultaneously: a feature extractor (the backbone) that maps raw inputs to a latent embedding space, and a classifier (the final linear layer) that draws decision boundaries between those embeddings.

Because both components are updated at the same time, they chase a moving target. If a mini-batch contains an imbalance of classes, or if the labels are noisy, the classifier's weights drift. This drift forces the feature extractor to constantly warp its layout, often leading to representation collapse, sensitivity to batch composition, and catastrophic forgetting.

A powerful alternative paradigm eliminates this co-optimization struggle entirely: fixed geometric classifiers. Instead of learning the final layer's weights, researchers pre-configure them into a rigid, mathematically optimal geometric structure that remains frozen throughout training. By turning the classifier into an unyielding anchor, the network is forced to adapt its internal representations to a stable, perfectly separated coordinate system.


The Geometry of Maximum Separation

To understand why a fixed classifier works, we must look at where deep networks naturally want to go.

During the terminal phase of standard training—when training error drops to zero—networks undergo a phenomenon known as Neural Collapse. As proven by Zhou et al., the optimization landscape naturally drives class features to cluster tightly around their respective class means, and these means align perfectly with the classifier's weight vectors.

Crucially, this final geometry forms a Simplex Equiangular Tight Frame (ETF).

   Standard Training: Moving Targets          Fixed Geometric Classifier

     Class A   Class B   Class C                 Class A   Class B   Class C
        \         |         /                       \         |         /
         \        |        /                         \        |        /
       [ Learnable Classifier ]                    [ Fixed Simplex ETF Anchor ]
              ▲       ▲                                       ▲
              │       │                                       │
     (Both layers drift and warp)               (Only the backbone optimizes)
              │       │                                       │
     [   Feature Extractor    ]                    [   Feature Extractor    ]

A Simplex ETF is the geometric equivalent of a perfectly balanced structure—like an equilateral triangle in 2D or a regular tetrahedron in 3D. In a Simplex ETF, every single class vector is maximally separated from every other vector, sharing the exact same negative inner product.

Since this perfectly symmetrical geometry is the theoretical end-state of successful training, frameworks like those proposed by Zhou et al. skip the process of learning it. By pre-defining the final layer as a fixed Simplex ETF, the model reduces its trainable parameter count and provides a clean, unchanging geometric blueprint for the feature extractor from step one.


Decoupling Optimization and Preventing Collapse

Why does fixing the classifier improve performance? The primary benefit is the decoupling of feature learning from class distributions.

In open-world semi-supervised learning and generalized category discovery, models must handle both known and completely novel categories. When classifiers are learnable, the prototypes for novel categories often drift or collapse into the territories of dominant, known classes.

To solve this, Wang et al. developed TRAILER, a framework that pre-assigns rigid Simplex ETF coordinates to both known and unseen categories. Because the targets are fixed and maximally separated, novel categories have a dedicated, stable territory in the embedding space, preventing representation collapse. Similarly, Luo et al. utilize fixed ETF category centers to eliminate the optimization instabilities that plague traditional clustering methods.

This stabilization applies to unsupervised and multi-view settings as well. For instance, Zhang et al. demonstrate that forcing latent representations to align with a pre-calculated, geometrically ideal set of anchors prevents data clusters from merging or dissolving during iterative optimization.


A Rigid Shield Against Noise and Imbalance

When training data is heavily imbalanced (long-tailed) or corrupted by noisy labels, learnable classifiers easily overfit. The optimizer warps the decision boundaries to accommodate mislabeled samples or expands the territory of majority classes at the expense of minority ones.

A fixed geometric classifier acts as an unyielding regularizer against this behavior. Because the optimizer cannot adjust the relative positions of the class anchors, the network cannot "cheat" by expanding a decision boundary. Instead, the feature extractor is forced to find robust, invariant features that map cleanly to the predefined targets.

  • Inductive Bias: Kasarla et al. show that introducing a fixed, closed-form matrix multiplication that guarantees maximum class separation before the softmax layer acts as a powerful regularizer, significantly boosting long-tailed recognition.
  • Label Noise Robustness: Zhou et al. introduce Prototype-Anchored Learning (PAL), demonstrating that fixed configurations prevent minority classes from collapsing into majority ones when annotations are imperfect.
  • Contrastive Stability: Gill et al. integrate non-trainable class vectors directly into supervised-contrastive loss functions, ensuring that the underlying feature space maintains a clean, symmetric geometry even under severe data imbalance.

Stationarity: The Key to Lifelong Compatibility

In real-world systems like image retrieval or face recognition, models are updated frequently. Normally, upgrading a model architecture requires a "re-indexing nightmare": reprocessing millions of gallery images because the new model's feature space is completely rotated or scaled compared to the old one.

Fixed geometric classifiers introduce stationarity—a permanent coordinate system for feature mapping.

   Traditional Upgrades: Space Warping           Stationary Upgrades: Hot-Swapping

     Model v1 Space      Model v2 Space            Model v1 Space      Model v2 Space
        (Class A)          (Class B)                  (Class A)          (Class A)
          ●   ◌              ◌   ●                      ●   ◌              ●   ◌
           \ /                \ /                        \ /                \ /
            ●                  ●                          ●                  ●
        (Class B)          (Class A)                  (Class B)          (Class B)

     [Incompatible Embeddings]                     [Directly Compatible Embeddings]

As demonstrated by Biondi et al. and Del Bimbo et al., if two entirely different architectures (e.g., a ResNet and a Vision Transformer) are trained using the exact same fixed d-simplex ETF classifier, they will map identical semantic concepts to the exact same regions of the representation space. This mathematical consistency allows developers to "hot-swap" models seamlessly without re-indexing existing databases.

This anchoring effect is equally vital for continual and incremental learning. In these scenarios, models must learn new tasks sequentially without forgetting old ones. He et al. (with DYSON) and Roy et al. pre-allocate class prototypes in a fixed geometric layout from the start. Because the territories for future classes are pre-reserved and old territories are locked, new class embeddings are prevented from drifting into or overwriting previously learned representations.


Generalization and Out-of-Distribution Detection

By enforcing strict angular separation, fixed geometric classifiers help models ignore spurious correlations (like background textures) and focus on core semantic features, improving out-of-distribution (OOD) generalization.

Chen et al. show that fixing a classifier as an ETF forces features from diverse, unseen environments to converge toward the same semantic prototypes. In semantic segmentation, Khandelwal demonstrates that replacing learnable weights with a fixed ETF maximizes the angular separation between visually similar classes (such as roads and sidewalks), significantly reducing domain confusion.

Furthermore, this structured space provides a reliable mechanism for identifying completely unknown data. In standard networks, OOD data often accidentally triggers high-confidence predictions because the classifier's decision boundaries extend infinitely.

To combat this, Xia et al. introduce Vast Predefined Classifiers (VPC). By pre-specifying an optimal feature geometry based on neural collapse principles, they allocate a dedicated, fixed geometric subspace for OOD features to occupy. Known classes remain locked in their equiangular territories, while unknown data naturally flows into the empty, predefined spaces between them, making anomalies clean and easy to detect.

Key Foundational Work

FoldHSphere: deep hyperspherical embeddings for protein fold recognition (2021) — contributed 1 concept elements to this direction.
Orthogonal Projection Loss (2021) — contributed 1 concept elements to this direction.
Hyperspherical Prototype Networks (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_geometric_classifier_6a08ffa2,
  title = {fixed geometric classifier},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-geometric-classifier/rdr_6a08ffa2884d4279b01700c36958bd8e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_fixed_geometric_classifier_6a08ffa2_v1,
  title = {fixed geometric classifier},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-geometric-classifier/rdr_6a08ffa2884d4279b01700c36958bd8e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake