TL;DR
Conic classification heads are designed to define bounded regions in feature space for better open-set recognition, but this paper proves that training them with standard softmax cross-entropy often fails to actually confine features. The authors introduce a new "asymptotic margin" metric that determines feature confinement, showing that standard training naturally pushes features toward infinity even when individual class regions are theoretically bounded.
What Problem Does This Paper Address?
In open-set recognition, a model needs to not only classify known categories but also "reject" unknown ones. A popular design choice is the conic classification head, which replaces standard linear decision boundaries with curved, bounded regions (like ellipses). The intuition is simple: if a feature falls outside these bounded "bubbles," it's likely an unseen class.
However, deep networks are almost always trained using softmax cross-entropy. This loss function is relative—it only cares about the differences between class scores. The authors identify a critical gap: just because a single class has a bounded score region doesn't mean the difference between scores stays bounded. If all class scores grow as you move away from the center, can the training process still push features to infinity to minimize loss? This paper provides the mathematical framework to answer that question.
Methods
The authors analyze a "norm-linear" classification head where the score for class $k$ is defined as: $$g_k(z) = \langle w_k, z - a_k \rangle + \alpha_k \|z - a_k\|_\epsilon$$ Here, $a_k$ is the class center (apex), $w_k$ is an axis vector, and $\alpha_k$ governs the "aperture" or how fast the score grows with distance.
Key methodological contributions include:
- Geometric Characterization: Proving that the regions defined by these scores are quadrics of revolution (ellipsoids, paraboloids, or hyperboloids) depending on the ratio of the linear term to the norm term.
- The Asymptotic Margin ($M_k$): The authors define a new metric that compares the growth rate of a class's score against all other classes in every possible direction. This is computed using a second-order cone program.
- Loss Surface Analysis: They characterize the infimum of the softmax cross-entropy loss. They show that if the asymptotic margin is positive, the loss can only reach zero as the feature norm goes to infinity.
Datasets
The theoretical results are validated using:
- CIFAR-10
- CIFAR-100
The models used are ResNet-18 architectures modified with various conic and prototype heads.
Key Insights and Ideas
- Boundedness is Relative: A class might have a bounded acceptance region in isolation, but if other classes grow even faster in the same direction, the softmax probability can still be driven to 1 by moving the feature to infinity.
- The "Aha" Moment on Confinement: Feature confinement is not a property of a single class's geometry; it's a global property of the entire head. The "asymptotic margin" captures this competition between classes.
- Softmax is a "Pusher": Softmax cross-entropy inherently rewards "over-confidence." Even with conic heads, the optimizer finds directions (rays) where it can push features to infinity to drive the loss closer to zero.
- Implicit Regularization: Even though features aren't strictly confined to a ball, conic heads act as a form of implicit regularization. They significantly reduce the average norm of features compared to standard linear heads, likely because the "reward" for growing features is weaker.
Key Results
- Universal Unboundedness: In experiments on CIFAR-10 and CIFAR-100, all 1210 class cells across 67 trained conic heads had positive asymptotic margins. This means that in every case, the features were not confined and escaped to infinity during training.
- Norm Reduction: Despite the lack of strict confinement, conic heads reduced the mean feature norm by 12% to 38% on CIFAR-10 compared to standard affine (linear) heads.
- Competitive Accuracy: Conic heads matched the accuracy of standard heads on CIFAR-10 and were only about 1 percentage point less accurate on CIFAR-100, proving they are viable alternatives for standard classification tasks.
- Theory vs. Practice: The "prototype head" (a special case where the linear term $w_k = 0$) behaved exactly as the theory predicted, keeping 9 out of 10 cells bounded.
Limitations and Future Work
- Analysis vs. Design: The paper focuses on analyzing why current conic heads behave the way they do under softmax, rather than proposing a new "SOTA" classifier.
- The "Why" of Norm Reduction: While the paper measures a significant reduction in feature norms, the exact mechanism of how the conic head weakens the reward for feature growth (compared to affine heads) is flagged for further study.
- Open-Set Performance: The authors note that for a bounded region to truly act as a confinement/rejection mechanism, the decision rule must use the class score directly (thresholding), rather than relying solely on the softmax output.
Summary updated