Low-Dimensional Hyperbolic Label Heads for Extreme Classification

In extreme multi-label classification, a model tags each document with a few relevant labels out of hundreds of thousands or millions. The final layer is one weight vector per label. With 500,000 labels and a 768-dimensional encoder, that layer has 384 million parameters. It dominates GPU memory, and in data-parallel training it dominates the gradient traffic that workers exchange every step. Current fixes keep the full-width layer and squeeze its storage, for example with 8-bit floats, or replace it with hard label trees that lose gradient flow when a routing decision goes wrong. This note proposes a different lever: shrink the per-label dimension itself, from hundreds to roughly 16–64. Hyperbolic space is a negatively curved geometry in which the room available grows exponentially with distance from the center, so it can hold large label hierarchies in very few dimensions. Each label is stored as a small hyperbolic point, and documents are scored by negative hyperbolic distance. The head shrinks by more than an order of magnitude, and its synchronization cost shrinks with it, with no compression step and no hard routing.

The obvious version of this idea has a trap, and the trap is where the research lies. Hyperbolic points are usually stored as coordinates inside a unit ball. Specific, rare "leaf" labels sit near the ball's edge. There, the learning signal reaching a label is a product of two small factors: its predicted probability, which is small for rare labels under the classification loss, and a geometric shrink factor that makes coordinate updates tiny near the boundary. Under the low-precision storage that extreme classification needs, those tiny updates round to zero. The labels that hyperbolic geometry is supposed to help most are the ones that stop learning. We propose storing each label as a tangent vector at the ball's center, whose length grows linearly with hyperbolic depth, and computing distances from that representation. The aim is that tail labels keep usable gradients in BF16 or FP8. A side benefit is that the stored parameters become ordinary Euclidean tensors, so off-the-shelf gradient compressors can be applied to them correctly.

Core Claim

The claim has three parts. First, a hyperbolic distance head with $d_h \approx 32$, stored in tangent coordinates and kept in BF16, can match a full-width Euclidean head on Precision@k for Wiki-500K and Amazon-670K, and do better on tail-label bins. Second, it cuts head memory and per-step head synchronization volume by about $d/d_h$ (roughly 24× at $d=768$), on top of what precision tricks give. Third, the usual ball-coordinate parametrization fails specifically on tail labels at low precision, through the compounding of two shrink factors described below.

To make the mechanism concrete, let $h$ be the document's encoder output mapped into the Poincaré ball (the unit-ball model of hyperbolic space), $w_c$ the ball point for label $c$, $p_c$ the model's predicted probability for $c$, and $y_c \in \{0,1\}$ its target. Under Riemannian gradient descent the update to $w_c$ is

$$ \Delta w_c \;\propto\; (p_c - y_c)\,\frac{(1-\|w_c\|^2)^2}{4}\,\nabla_{w_c} d_{\mathbb{H}}(h, w_c), $$

where $d_{\mathbb{H}}$ is hyperbolic distance and the middle factor is the inverse metric used by Poincaré Embeddings for Learning Hierarchical Representations. The Euclidean distance gradient grows roughly like $1/(1-\|w_c\|^2)$ near the edge, so the net coordinate step scales like $(1-\|w_c\|^2)$. In exact arithmetic this is geometrically correct. In BF16, however, a coordinate near 0.99 has a spacing of about $4\times 10^{-3}$ between representable values, so steps on deep labels vanish.

The tangent parametrization stores $v_c = \log_0(w_c)$, the label's tangent vector at the center, whose norm is half the label's hyperbolic distance from the center. Moving a deep label by a fixed hyperbolic distance then needs a coordinate change of fixed size, which BF16 represents without trouble.

Relation to Prior Work

Two papers motivate the dimension argument. Poincaré Embeddings for Learning Hierarchical Representations learns hierarchy embeddings in the Poincaré ball and reports a mean average precision (MAP) of 0.823 with 5 dimensions on WordNet nouns, against 0.168 for a 200-dimensional Euclidean embedding. Hyperbolic Neural Networks supplies hyperbolic classification layers and exponential/log maps. Neither works with supervised extreme label spaces or low-precision storage. The WordNet result concerns reconstructing a known hierarchy, not classifying documents, so whether it transfers is exactly what we test.

The closest prior system is Hyperbolic Capsule Networks for Multi-Label Classification (HYPERCAPS). It routes text features to hyperbolic label capsules and prunes candidate labels with a gating heuristic. It reaches P@1 of 91.42 on EUR-Lex-57K but stops at 4,271 labels. It does not treat the head's parameter count, precision, or synchronization as the object of study, and it does not examine tail-label gradient decay. We differ in scale, more than 100× more labels, and in treating storage and gradient flow as the central questions.

Several methods constrain the design without addressing dimension. ELMO keeps a flat full-width head in FP8 with stochastic rounding and fused kernels, cutting Amazon-3M peak memory to 6.6 GiB from 39.7 GiB for Renee. Its need for stochastic rounding to stop small updates rounding to zero is direct evidence that the precision failure we predict is real. Because ELMO compresses each entry rather than reducing the number of entries, its approach can be stacked with ours. Softmax Tree and X-Transformer get sublinear cost from hard trees or label clusters. A wrong route there gives the true label zero probability. Our head stays dense and differentiable over every label and only cuts its width. Dynamic sparse training for 670k labels prunes connections instead, but needs a large dense intermediate buffer to avoid underfitting.

The softmax-starvation literature supplies the diagnostic lens. Syntax vs. Semantics and Eureka-Moments in Transformers show that a gradient scaled by a small probability lets dominant signals starve weak ones. Disentangled Non-local Neural Networks shows that a multiplicative product of factors suppresses gradients whenever one factor is small. We carry that product view into the output head, where the second factor is geometric rather than attention-based.

On communication, Fast Parallel Training of Neural Language Models finds that the output layer drives up to half of synchronization time, and samples rows to reduce it, which leaves tail rows with stale gradients. DeMo and BitDP compress Euclidean gradients, and BitDP explicitly avoids aggressive quantization of embedding and output heads. On the Utility of Gradient Compression warns that runtime compression competes with backward-pass compute. Shrinking the head structurally avoids that compute contention entirely.

Research Program

The build is a pretrained Transformer encoder, kept Euclidean, followed by a linear map to $\mathbb{R}^{d_h}$ and an exponential map into the ball. Labels are stored as BF16 tangent vectors $v_c$. Logits are $-\beta\, d_{\mathbb{H}}(h, \exp_0(v_c)) + b_c$, with a learned scale $\beta$ and per-label bias $b_c$. Training uses a binary loss over positives plus shortlisted hard negatives, computed in chunks as in ELMO so the full logit matrix is never materialized. The curvature is learned, so the method can approach the Euclidean case if hierarchy does not help.

The analysis tracks per-label update magnitudes and the fraction of updates that round to zero, binned by label frequency. This tests the compounding-attenuation hypothesis directly, comparing ball and tangent storage at FP32, BF16, and FP8. The distributed component runs 8-way data parallelism and logs head bytes per step and wall-clock time. At Wiki-500K scale the head is about 1.5 GB of FP32 gradient per step at $d=768$, against about 32 MB at $d_h=32$ in BF16. The same runs check whether DeMo-style compression applied to tangent-coordinate heads trains stably. The second objective is to show that this works where ball-coordinate heads diverge.

Baselines

The naive reference is the standard full-width (768-dimensional) FP32 linear head trained with ordinary data parallelism. It fixes the accuracy target and the uncompressed memory and communication cost.

The strongest accuracy competitor is X-Transformer, reported at P@1 of 77.28% on Wiki-500K (X-Transformer), above AttentionXML at 76.95% and Parabel at 68.70%. On EUR-Lex-57K the reference is HYPERCAPS at P@1 91.42. For Amazon-670K there is no reported result found on Lacuna for P@1 of any baseline; the file states only that sparse training stays within 1–2% of dense. We will therefore rerun our own dense head there. Beating X-Transformer is not required. The bar is to come within one point of P@1 using the same encoder with a 24× smaller head, and to exceed it on tail-frequency bins.

The strongest memory competitor is ELMO with an FP8 full-width head, run on the same encoder, shortlist, and epoch budget. Beating it means lower peak memory at equal P@k, and lower memory still when our head is also stored in FP8.

The strongest communication competitors are BitDP and DeMo applied to the full-width head, and Xiao et al.'s row sampling. All are tuned with the same hyperparameter search budget as our method. Beating them means fewer head bytes per step and equal or better tail P@k, with no added compression compute.

The ablation that isolates the key ingredient is the identical hyperbolic head in ball coordinates trained with Riemannian Adam, at matched precision. A second ablation is a Euclidean bottleneck head at the same $d_h$, both with plain dot products and with normalized cosine logits.

Why not just…?

Why not just use a low-dimensional Euclidean bottleneck head? This is the most serious threat. If a 32-dimensional dot-product or cosine head matches ours, then the savings come from low rank, not from curvature. The Poincaré result suggests hierarchy needs far fewer dimensions in hyperbolic space, but that evidence comes from hierarchy reconstruction. The matched-$d_h$ sweep (16, 32, 64, 128) on Wiki-500K, Amazon-670K, and the hierarchical EUR-Lex-57K and CIE-10-ES settles it. If Euclidean heads match at every width, we report that the hyperbolic claim fails and keep only the low-rank memory and communication finding. We expect the gap to be largest on tail bins and on explicitly hierarchical label sets.

Why not just quantize the full head with ELMO? ELMO shrinks bytes per parameter but keeps $C\cdot d$ parameters. It is single-node, so data-parallel traffic for the head is still the full gradient. Our method is complementary; the test is whether FP8 tangent storage at $d_h=32$ beats FP8 at $d=768$ on memory and traffic without losing P@k.

Why not just compress the head's gradients with BitDP or DeMo? BitDP itself avoids aggressive quantization of output heads, and Agarwal et al. show compression costs compute. The experiment compares head bytes, wall-clock time, and tail P@k for compressed full-width heads against an uncompressed small head. We also test compression stacked on the small head, which tangent storage makes legitimate.

Why not just train the ball-coordinate head in FP32 with Riemannian Adam? In FP32 the precision failure may not appear, and adaptive step sizes partly cancel the shrink factor. That is why the ablation spans FP32 to FP8. If ball and tangent storage perform the same in BF16 and FP8, the attenuation hypothesis is wrong and tangent storage is merely a convenience.

Decisive Checks

Three measurements decide the proposal. First, rounding-to-zero rates and update norms by label-frequency bin: the hypothesis predicts collapse for deep tail labels in BF16 ball storage and not in tangent storage. Second, matched-width hyperbolic versus Euclidean heads on tail-bin P@k, plus the Information Contrast Model metric on CIE-10-ES from Evaluating Extreme Hierarchical Multi-label Classification, which rewards predicting labels that are close in the hierarchy. Third, end-to-end head bytes and step time under 8-way data parallelism against ELMO, BitDP, and DeMo baselines. A positive result requires all three to point the same way.