In traditional supervised learning, a neural network is a prisoner of its own architecture. When we train a classifier to distinguish between cats and dogs, we typically hard-code a final linear layer with exactly two output nodes. This design creates a "fixed label space"—a rigid mapping where the model learns to associate internal representations with specific, static indices.
As noted by Xu et al., this architectural rigidity is a fundamental bottleneck. If you want to add a third class, such as "birds," you cannot simply update the model; you must modify the network structure and retrain. This constraint hinders progress in continual learning, zero-shot transfer, and open-world applications where the set of categories is constantly evolving.
The Softmax Trap and Architectural Rigidity
The core of the problem lies in the final layer. Standard models use a softmax function over a fixed number of channels or neurons. Aljundi et al. et al. observe that this static structure makes it difficult for models to accommodate new classes dynamically over time. In semantic segmentation, for instance, Li et al. point out that coupling the output head to a predefined taxonomy prevents the system from adapting to new categories without structural changes.
This rigidity isn't just a matter of convenience; it limits the model's intelligence. Nandwani et al. argue that treating output values as discrete, fixed indices prevents models from learning the underlying relational logic between those values. A model trained to solve a 9x9 Sudoku board fails on a 16x16 board because its output space is physically tied to the number 9.
Classification as Retrieval and Similarity
To break this bottleneck, researchers are moving away from index-based prediction toward metric learning and retrieval. Instead of asking "Which index does this input belong to?", the model asks "Which known example is this input most similar to?"
Wiseman and Stratos pioneered this with "label-agnostic" sequence labeling. By learning a similarity function between word contexts rather than learning specific categories, their model can transfer to entirely new label sets without retraining. Similarly, Gominski and Gouet-Brunet reframed remote sensing classification as an image retrieval problem, allowing a model to recognize novel land-use types by comparing them to a dynamic database of embeddings.
By treating classification as a series of binary decisions against a dynamic "memory" of class embeddings, as proposed by Mojzísek and Hula, systems can reject unknown classes and incorporate new ones seamlessly.
Labels as First-Class Citizens
Another powerful strategy is to move labels from the output of the network to the input. In this paradigm, the model is conditioned on the label it is looking for.
- Textual Conditioning: Chochlakis et al. developed Demux, which treats emotion labels as input tokens. This allows the model to predict unseen emotion types simply by changing the input prompt.
- Visual Conditioning: In medical imaging, Ma et al. introduced Label Conditioned Segmentation (LCS). Instead of outputting dozens of channels for different organs, the model outputs a single channel conditioned on a specific "atlas" label, enabling it to segment structures it was never explicitly trained to recognize.
By mapping both inputs and labels into a shared embedding space—an approach used by Li et al. in their LSeg architecture—models can perform "zero-shot" tasks using arbitrary label sets at test time.
Reframing Classification as Reasoning
The most recent shift involves treating classification as a natural language reasoning task. Rather than mapping a sentence to a "Location" tag, Cui et al. use a generative model to score templates like "Bangkok is a location entity." This reframes Named Entity Recognition (NER) as a language model ranking problem, bypassing the need for fixed classification heads.
Similarly, Huang et al. reframe NER as a machine reading comprehension task, while Xu et al. reformulate multi-class classification into a binary instruction-following task. In these systems, the "label space" is as broad as natural language itself.
Toward Dynamic Architectures
For tasks that still require specialized layers, researchers are exploring ways to generate them on the fly. Bansal et al. introduced LEOPARD, a meta-learning framework that generates task-specific classification layers for each new problem. This allows the model to adapt to tasks with varying numbers of classes without structural modification.
By moving away from fixed output heads, these diverse approaches share a common goal: transforming neural networks from rigid index-predictors into flexible reasoners capable of navigating an open and ever-changing world of concepts.
Key Foundational Work
Query-Driven Multi-Instance Learning (2020) — contributed 1 concept elements to this direction.
Label-Agnostic Sequence Labeling by Copying Nearest Neighbors (2019) — contributed 1 concept elements to this direction.
Task-Free Continual Learning. (2019) — contributed 1 concept elements to this direction.