Conic Structure of the Cross-Entropy Loss at the Final Layer and Its Absence in Rectifier Backbones

Omer Nezih GerekarXiv 2026 · arXiv

TL;DR

This paper provides a mathematical explanation for why deep learning loss surfaces often appear as single, smooth basins with nearly straight walls. It proves that at the final layer, the loss is essentially a polyhedral cone with a predictable "core radius," while in the backbone, the millions of tiny "kinks" caused by ReLU activations are too small and scattered to form significant creases, preserving the overall smooth appearance of the landscape.

What Problem Does This Paper Address?

When researchers visualize the "loss landscape" of a neural network—usually by taking a 1D or 2D slice through parameter space—they often see a characteristic "bowl" shape. Interestingly, the walls of these bowls often look remarkably straight far from the minimum, resembling a cone.

This raises a fundamental geometric question: Is the loss surface a single cone that is simply rounded at its tip (the apex), or is it a "lower envelope" of many different cones that meet at sharp, jagged edges (creases)? Understanding this geometry is crucial because "creases" would imply sudden, large jumps in the gradient, which could affect how optimization algorithms like SGD behave. The authors aim to move beyond visual intuition to provide exact mathematical statements about the shape of the softmax cross-entropy loss.

Methods

The authors split their analysis into two parts:

  1. The Final Layer: They analyze the loss as a function of the weights in the final affine layer (where logits depend linearly on weights). They use convex analysis and the concept of a recession function—a tool that describes how a function grows as you move infinitely far in a specific direction.
  2. The Backbone: They analyze the loss as a function of the parameters in the "backbone" (the layers before the final one), where signals pass through rectifiers (ReLUs). Here, they employ the dimension theory of semialgebraic sets. Since ReLU networks are piecewise polynomial, the points where a neuron switches from "off" to "on" are roots of polynomials. They use this to count how many samples "switch" at any given point in parameter space.

Datasets

The theoretical results are illustrated with experiments on:

  • CIFAR-10: A standard image classification dataset.
  • Architectures: The authors used ResNet-18 (for final layer measurements) and ResNet-56 (for backbone measurements, testing versions both with and without skip connections).

Key Insights and Ideas

  • The "Core Radius" of Confidence: At the final layer, the loss isn't just a cone; it's a "rounded" cone. The authors define a core radius—the distance from the center where the walls become effectively straight. Crucially, they show this radius is proportional to the mean surprisal of the classes. If a model is very confident at a certain point, the core radius is large, meaning the "flat" or rounded part of the basin is wider.
  • Kinks vs. Creases: In a ReLU network, every time a single neuron for a single image switches its activation state, the loss surface has a "kink" (a discontinuity in the derivative). You might expect millions of these kinks to bunch up and form a sharp "crease." However, the authors prove that under a "coprimality condition" (essentially, that images are algebraically distinct), these kinks almost never happen at the same place.
  • The 1/n Share: Because kinks are scattered, each one only carries the "share" of a single sample ($1/n$). In a dataset with 50,000 images, a single kink is tiny. This explains why the loss surface looks smooth to the human eye—the "wrinkles" are there, but they are too small to form the sharp ridges (creases) that would characterize a multi-cone envelope.

Key Results

  • Final Layer Exactness: Theorem 2.1 proves the final-layer loss stays within a band of constant width around a polyhedral cone. Theorem 2.2 provides an exact expansion showing the loss approaches this cone exponentially fast as you move away from the center.
  • Backbone Smoothness: For ResNet-56, the authors measured the "roughness" of the loss surface. They found that any "hidden" creases are bounded to be less than 2.5% of the local slope for networks with skip connections (and even less for those without).
  • Visual Confirmation: Slices of ResNet-18 and ResNet-56 confirmed that the "single basin" view is accurate. Even in "plain" networks (without skip connections), which are often described as having chaotic loss surfaces, the authors found a single basin, though it was much steeper and more lopsided than in ResNets.

Limitations and Future Work

  • Distribution Assumptions: The proof regarding the backbone (Theorem 3.3) assumes directions are drawn from a distribution with a density. It doesn't strictly cover "filter-normalized" directions (a common visualization technique), though experimental evidence suggests the results still hold there.
  • Algebraic Coincidences: The "coprimality condition" could fail if a dataset has many identical images or if images have large, perfectly flat saturated regions (like pure black or white patches), which might cause kinks to align and form larger creases.
  • Non-linear Heads: The analysis of the final layer assumes a standard affine head. It does not cover more exotic architectures like "conic heads" where scores are non-linear.

Overall, the paper provides a rigorous foundation for the "common wisdom" of loss surface visualization, proving that the "smooth bowl" we see is a mathematically sound representation of the underlying optimization landscape.

Summary updated

Abstract

Slices of the loss along lines and planes in parameter space are widely used to visualize deep networks. They often show a single basin with nearly straight walls, which raises the question of whether such a loss is a cone rounded at its apex, or a lower envelope of several cones that meet along creases. We answer this question for softmax cross-entropy at two levels of a classifier. Over the weights of an affine final layer, the loss equals a polyhedral cone plus an offset that decreases monotonically and exponentially along every ray, towards a limit in closed form. The distance at which the asymptotic wall reaches the height of the starting point, which we call the core radius, equals the mean surprisal of the classes that the direction favours, divided by the slope of the wall. Over the parameters of a rectifier backbone, the loss along a line has a kink wherever the activation pattern of a sample changes. Using the dimension theory of semialgebraic sets, we show that, along almost every line, the samples that switch at one point share an irreducible factor of their pre-activation polynomials. Under a coprimality condition, each kink therefore carries the share of one sample among $n$, so creases comparable to the slope do not form. Measurements on ResNet-18 and ResNet-56 trained on CIFAR-10 illustrate both results, with a test that bounds the size of any crease that sampled loss values can hide.
BibTeX
@misc{gerek2026conic,
  author = {Omer Nezih Gerek},
  title = {{Conic Structure of the Cross-Entropy Loss at the Final Layer and Its Absence in Rectifier Backbones}},
  year = {2026},
  eprint = {2611.20696},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20696},
}
Figure