When a deep learning classifier is trained on a closed set of categories, it maps out a high-dimensional embedding space where every region is aggressively partitioned. If the model only knows "cats" and "dogs," any input falling into the intermediate space between them will still be assigned to one of those two categories—often with dangerously high confidence.
To prevent this overconfidence in open-set, out-of-distribution (OOD), or incremental learning scenarios, a model must learn what it does not know. However, obtaining real-world data for every possible unknown category is impossible.
The core insight of this research direction is that the latent space between known classes can be used to hallucinate novel ones. Geometrically, many unseen categories reside within the convex hull of known base classes (Jeong et al.). By interpolating between the representations of distinct known classes, researchers can populate these low-density "gaps" with synthetic proxy classes, forcing the model to learn smoother, more realistic decision boundaries.
Mechanisms for Hallucinating Classes
Instead of mixing raw pixels or inputs—which often introduces disruptive visual artifacts—this paradigm operates within the hidden layers or generative latent spaces of neural networks.
[Class A Representation] ----(Interpolation/Mixup)----> [Synthetic Proxy Class] <----(Interpolation/Mixup)---- [Class B Representation]
1. Manifold Mixup in Hidden Layers
Rather than relying on external data, models can generate synthetic outliers internally during the forward pass. Da-Wei Zhou et al. demonstrate this by applying linear interpolation to the internal feature representations of distinct known classes. These blended features are assigned to a dedicated "placeholder" class.
Because these virtual instances occupy the low-density regions between established clusters, they serve as highly effective proxies for out-of-distribution data. Zifeng Cheng et al. and Qin Zhang et al. leverage this exact mechanism to train classifiers to recognize open-set inputs without ever exposing them to real anomalous data during training.
2. Generative Latent Space Mixing
When structural coherence is paramount, generative models can guide the interpolation. Zongyuan Ge et al. show that mixing known class embeddings within the latent space of a Generative Adversarial Network (GAN) produces structurally sound, domain-specific samples that do not belong to any single training class.
Similarly, Parsa Rahimi et al. utilize a conditional diffusion framework to interpolate class labels, generating entirely new synthetic identities that expand the diversity of the training set.
3. Distributional Parameter Mixing
Instead of interpolating individual data points or features, some approaches operate on the underlying geometry of the classes themselves. Chaofan Chen et al. propose mixing the mean and covariance parameters of existing class distributions. This captures the structural variance of the dataset, allowing the model to simulate entirely new class distributions that maintain realistic statistical properties.
Why It Works: Structuring the Embedding Space
Synthesizing proxy classes works because it fundamentally alters how the model structures its embedding space. It shifts the model from a strategy of global partitioning (carving up the entire space among known classes) to one of localized clustering.
- Carving Out "No Man's Land": By introducing synthetic features into the gaps between classes and labeling them as "unknown," the model learns to draw tight decision boundaries around known categories. Soroush Seifi et al. demonstrate that this strategy trains the network to treat intermediate regions as non-class territory. To prevent these synthetic points from accidentally diluting real class clusters, Tobias Koch et al. introduce distance-based filters that force synthetic unknowns away from known class centroids.
- Smoothing the Metric Space: In deep metric learning, the authors of Proxy Synthesis linearly interpolate both image embeddings and their corresponding class proxies. This forces the network to optimize for a continuous, smooth transition across the embedding space, which significantly improves generalization to genuinely unseen categories during inference.
- Proactive Space Reservation: For incremental learning tasks where new classes arrive over time, a model typically suffers from catastrophic forgetting or representation drift. Da-Wei Zhou shows that interpolating existing features allows a model to proactively practice mapping novel patterns to reserved, unallocated regions of the embedding space before the actual new data is ever encountered.
Generalization Across Modalities
The principle of cross-class feature interpolation is highly versatile and generalizes across diverse domains:
- Computer Vision & Robotics: Hyeonseong Kim et al. use manifold mixup to blend features in LiDAR semantic segmentation, training autonomous vehicles to cluster and discover real-world unknown objects on the fly.
- Speech & Natural Language Processing: In spoken language recognition, Rong Tong et al. mix phonotactic n-gram features across languages to establish robust open-set boundaries. For dialogue systems, Hao Lang et al. mix feature-level representations of known intents to detect out-of-domain user queries. Even in large language models, Wei He et al. generate intermediate synthetic demonstrations to bridge the gap when a user's prompt falls outside the scope of the provided examples.
- Graph Learning & Meta-Learning: Nan Yin et al. interpolate dissimilar source domain instances to simulate novel classes for open-set graph domain adaptation. In meta-learning, Huaxiu Yao et al. interpolate features across entirely different tasks and assign unique labels to the resulting synthetic classes, effectively "densifying" the task distribution and preventing the meta-learner from memorizing a sparse training set.
Key Foundational Work
Generative OpenMax for Multi-Class Open Set Classification (2017) — contributed 2 concept elements to this direction.
Virtual example for phonotactic language recognition (2014) — contributed 2 concept elements to this direction.