Hyperbolic space is a curved geometry whose volume grows exponentially with distance from the origin, much as the number of nodes in a tree grows with depth. That property lets a small encoder place a taxonomy of tens of thousands of concepts into a handful of dimensions. General concepts sit near the center and specific ones near the edge. Zeroth-order optimization trains a network using forward passes only. It nudges the weights along a random direction, measures how much the loss changes, and steps along that direction. Because nothing needs to be cached for a backward pass, it fine-tunes large language models at inference-level memory. These two ideas should combine naturally, for example to fine-tune a billion-parameter language encoder that places new scientific terms into a hyperbolic taxonomy on a single GPU. We argue that the naive combination fails for a specific, measurable reason.
A hyperbolic output layer stretches distances by a factor that grows exponentially with how deep a concept sits. The per-example loss changes that a zeroth-order step adds up are therefore dominated by the few deepest, most boundary-hugging examples in each batch. As a result, the scalar that drives every update is heavy-tailed. For leaf concepts it falls outside the regime where a finite difference approximates a derivative. For root concepts it is nearly zero. We propose a forward-only estimator that keeps each example's loss difference separately, which costs nothing extra because the forward pass already produces them. It then downweights each difference by the inverse of the local hyperbolic stretch at that example's embedding. This is the forward-only analogue of the rescaling that Riemannian SGD applies to hyperbolic gradients. It preserves the seed-based, inference-memory implementation of MeZO unchanged.
Core Claim
Consider an encoder with Euclidean weights $\theta$ that maps a term $x_i$ to a point $e_i$ in the Poincaré ball, the open unit ball in which hyperbolic distances blow up near the edge. Its loss on that example is $\ell_i(\theta)$. The standard two-point zeroth-order estimate averages the per-example differences $\ell_i(\theta+\epsilon z)-\ell_i(\theta-\epsilon z)$ with equal weight, where $z$ is a Gaussian direction and $\epsilon$ a small radius. We propose instead
$$\hat g = \frac{1}{B}\sum_{i=1}^{B} w_i\,\frac{\ell_i(\theta+\epsilon z)-\ell_i(\theta-\epsilon z)}{2\epsilon}\,z, \qquad w_i \propto \lambda_{e_i}^{-2} = \frac{(1-c\|e_i\|^2)^2}{4},$$
where $B$ is the batch size and $c$ the curvature magnitude. The factor $\lambda_{e}=2/(1-c\|e\|^2)$ is the conformal factor, meaning the local stretch of the hyperbolic metric at $e$. The weights are renormalized to mean one per batch so learning rates stay comparable. For the Lorentz model the weight is the inverse squared stretch of the map from encoder output to hyperboloid, which grows like $\sinh(r)/r$ at radius $r$.
The claim has three parts. The first is diagnostic: under the equal-weight estimator, a small fraction of deep examples carries most of the magnitude of the update scalar. The second is ZO-specific: this matters more for zeroth-order than for first-order training. With Gaussian directions the relative variance of the estimator depends only on dimension, not on geometry. The damage therefore does not come from idealized variance. It comes from three effects that backprop with Adam largely escapes. The update scalar is heavy-tailed across batches. MeZO-style SGD has no per-coordinate normalization to absorb a single huge scalar. And the finite-difference Taylor remainder, which scales with $\lambda^2$, biases the estimate for leaves. The third part is the fix: depth weighting removes most of the gap between hyperbolic and Euclidean heads that opens up under zeroth-order training, at equal forward-pass budget.
This differs from the closest prior work in where the geometry lives. Zeroth-order Optimization on Riemannian Manifolds samples perturbations in the tangent space of a manifold-valued parameter and maps them back with a retraction. Riemannian Zeroth-Order Gradient Estimation with Structure-Preserving Metrics for Geodesically Incomplete Manifolds stretches the metric near a boundary and uses rejection sampling so perturbations never leave the domain. Both treat the parameters themselves as points on a curved space. Both were tested only on synthetic objectives, robot stiffness matrices, adversarial attacks and meshes, never on hyperbolic representation learning. In the modern hyperbolic encoders we target, the weights are Euclidean and the curvature enters only through the output map. Examples include the Poincaré entailment-cone encoder of Geometric Constraints for Small Language Models to Understand and Expand Scientific Taxonomies and the radius adapter of HyperET.
This dissolves two problems the research file lists as open. Boundary crossing cannot occur when points are produced by an exponential map from the origin. Seed-based in-place perturbation from Fine-Tuning Language Models with Just Forward Passes works unchanged, because nothing non-linear is applied to the weights. What remains is the problem those papers could not see: curvature acting on the loss through the data rather than on the parameters.
Ma and Huang's result that zeroth-order error grows with curvature magnitude motivates us. Their bound concerns perturbations on the manifold, so we treat it as a hypothesis to check empirically in the pulled-back setting, not as a result we inherit. The intrinsic-dimension savings of Li et al. constrain us rather than help: our parameters are ambient Euclidean weights, so we claim no such savings.
Research Program
The first component is a measurement study on a model small enough to backpropagate. We use a DistilBERT-base encoder with a Poincaré or Lorentz head, trained with a distance-based hypernymy loss and, separately, with entailment cones as in SS-MONO. At checkpoints we compute three quantities per depth bucket and per norm decile.
The first quantity is each example's share of the squared update scalar. The second is the cosine between the zeroth-order estimate, averaged over many seeds, and the backprop gradient. The third is a linearity test that compares one-point and two-point differences to detect examples whose finite difference is dominated by curvature of the loss. We also log how often FP16 and BF16 forward passes return identical perturbed losses for shallow examples, and how often they return non-finite losses for deep ones. The research file's precision concerns, raised in Numerically Accurate Hyperbolic Embeddings Using Tiling-Based Models and Hyperbolic vs Euclidean Embeddings in Few-Shot Learning, predict both effects.
The second component is the estimator itself, built on the MeZO codebase (princeton-nlp/MeZO). The only change is that the forward pass returns the vector of per-example losses and embedding norms instead of their mean. Memory stays at inference level, since $B$ extra floats are negligible. We will study two variants. One uses norm-based weights, which require no labels. The other uses weights computed from the known taxonomy depth, which separates the effect of the geometry from the effect of the depth labels.
The third component scales the setup. We use OPT-1.3B and OPT-13B encoders under the same head, where full first-order fine-tuning does not fit on one 80GB A100 but MeZO does. We also test compatibility with the hybrid scheme of New Hybrid Fine-Tuning Paradigm for LLMs, with the backbone trained zeroth-order and the hyperbolic head trained with first-order Riemannian SGD. The research file frames that paper's smoothness condition, a large gradient Lipschitz constant for the base model and a small one for the adapter, as an open question when the adapter is hyperbolic. We measure the head's Lipschitz estimates directly instead of assuming the condition holds.
Baselines
The naive reference point is equal-weight MeZO on the encoder with the same hyperbolic head, same seeds, same $\epsilon$ grid and same number of forward passes. It is the method a practitioner would try first, and our diagnostics predict where it fails.
The strongest existing method for memory-constrained training is the hybrid zeroth-order-backbone and first-order-adapter scheme of New Hybrid Fine-Tuning Paradigm for LLMs. That paper reports reaching 90% SST-2 accuracy with OPT-1.3B in 2,500 steps against at least 12,500 for baselines, and winning in 41 of 54 comparisons against first-order PEFT. We run it with the hyperbolic head as the first-order-trained module, at matched peak memory and matched forward-pass count. We tune its two learning rates over the same grid size we give our method. Beating it means higher taxonomy metrics at equal memory and equal forward passes, or equal metrics in fewer steps. Our weighting applies to its backbone track, so we also report hybrid-plus-weighting.
The accuracy ceiling is first-order fine-tuning with Adam where memory allows. For the task itself that means SS-MONO from Geometric Constraints for Small Language Models. For SS-MONO's numbers on SemEval-Food, MeSH and WordNet-Verb there is no reported result found on Lacuna, since the file notes that tables exist but does not transcribe them. We therefore rerun it ourselves on DistilBERT under its published protocol. For zeroth-order parity, the reference is MeZO's report that OPT-13B tuned with MeZO was within 1% of Adam fine-tuning on 7 of 11 tasks (Fine-Tuning Language Models with Just Forward Passes). Our target is to restore that kind of parity on hierarchical tasks, where we expect the naive method to fall well short of it.
To separate a better objective from a better estimator, we train first-order Adam with the same $\lambda^{-2}$ per-example weights. If that variant gains as much as the zeroth-order version, the weighting is just a better loss, not a ZO fix. A Euclidean head trained with MeZO tests whether hyperbolic geometry survives zeroth-order training at all.
Two ablations isolate the key ingredient. The first applies the same weights randomly permuted within the batch, which keeps the weight distribution but removes its alignment with depth. The second is per-example clipping of loss differences at a tuned quantile, which is geometry-free robustness applied at the same granularity.
Why not just…?
Why not just train the hyperbolic model with first-order Riemannian SGD? For shallow embedding tables like the WordNet experiments of Poincaré Embeddings for Learning Hierarchical Representations, one should. There is no memory wall, so we do not propose zeroth-order training there. The case for forward-only training arises when a large language encoder produces the embeddings. There, backprop activation memory is the constraint MeZO documents, with OPT-30B fitting under MeZO on one A100 while Adam fits only 2.7B. The scaling experiments test exactly that regime, and the first-order ceiling is always reported where it fits.
Why not just freeze the backbone, train the hyperbolic head by backprop, or use the hybrid scheme? This is the most serious competitor. A frozen or zeroth-order backbone with a first-order hyperbolic head is cheap and avoids zeroth-order estimation on the head entirely. It does not avoid the problem we identify, though. Whenever the backbone is updated forward-only through a hyperbolic head, its update scalar still passes through the conformal factor. The hybrid paper's own evidence that updating the base model beats frozen-base PEFT is what makes backbone updates worth doing. If hybrid training without weighting matches hybrid-plus-weighting, our claim is false for this regime, and we will report it that way.
Why not just clip the update scalar, or clip each example's loss difference? Clipping the batch-level scalar treats a batch dominated by one leaf the same as a batch where all examples agree. It throws away the shallow examples' signal along with the outlier. Per-example quantile clipping is a stronger alternative and could plausibly match our weighting, because much of what we describe is heavy-tail robustness. Our method differs in using a weight fixed by geometry rather than a tuned threshold. It also downweights leaves smoothly even when they are not outliers, which we argue corrects the finite-difference bias for leaves. The per-example clipping ablation, tuned with the same budget, decides this. If it ties, the honest conclusion is that the hyperbolic failure mode is real but its cure is generic robust estimation.
Why not just use a high-dimensional Euclidean head trained with MeZO? The file's evidence argues strongly for hyperbolic geometry under first-order training. It reports 5-dimensional Poincaré embeddings reaching WordNet noun MAP 0.823 against 0.168 for 200-dimensional Euclidean embeddings. The SS-MONO ablation also reports a clear drop when the hyperbolic encoder is removed. Whether that advantage survives zeroth-order training is unknown. If naive MeZO erases it, a Euclidean head becomes the rational choice. Our experiment measures the hyperbolic-minus-Euclidean gap under first-order training, naive MeZO and weighted MeZO.
Proposed Experiments and Decisive Checks
The main benchmarks are SemEval-Food, MeSH and WordNet-Verb taxonomy expansion under SS-MONO's protocol and metrics. We add WordNet noun transitive-closure reconstruction scored by mean average precision and mean rank, using an encoder over term names rather than a free embedding table, and HyperLex graded entailment scored by Spearman correlation. The shallow-embedding figures give context but are not directly comparable, because an encoder must generalize across terms. They are Poincaré MAP 0.823 and HyperLex Spearman 0.512 from Poincaré Embeddings for Learning Hierarchical Representations, and hyperbolic VAE MAP 0.506 from A Wrapped Normal Distribution on Hyperbolic Space. All zeroth-order runs share the same total forward-pass budget, peak memory measured on one 80GB A100, and three seeds. We also report wall-clock time, which the research file cites as a known weakness of zeroth-order methods.
The first decisive check is the diagnosis. On DistilBERT, the top norm decile should account for a disproportionate share of the squared update scalar under naive MeZO. It should also show the lowest linearity-test pass rate. If deep examples carry roughly their proportional share, the premise fails and the project stops there.
The second check is ZO-specificity. The gain from weighting under zeroth-order training should clearly exceed the gain from the same weighting under first-order Adam.
The third check is gap closure. On all three expansion benchmarks, the hyperbolic-minus-Euclidean gap should shrink substantially from first-order to naive MeZO, and weighted MeZO should restore most of it. Weighted MeZO should also beat the permuted-weight and per-example-clipping ablations at matched tuning budget.
The fourth check is scale. At OPT-13B, weighted MeZO should outperform naive MeZO and the unweighted hybrid scheme at equal memory. Adding weighting to the hybrid backbone track should not hurt.
A positive result on the first three checks would show that curvature in the output map is a distinct and fixable failure mode of forward-only training. A negative result on the third, with the Euclidean head winning under zeroth-order training, would be a useful finding in its own right: memory-constrained practitioners should drop hyperbolic heads.
References
- Zeroth-order Optimization on Riemannian Manifolds
- Riemannian Zeroth-Order Gradient Estimation with Structure-Preserving Metrics for Geodesically Incomplete Manifolds
- Geometric Constraints for Small Language Models to Understand and Expand Scientific Taxonomies
- HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
- Fine-Tuning Language Models with Just Forward Passes
- Numerically Accurate Hyperbolic Embeddings Using Tiling-Based Models
- Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin
- New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework
- Poincaré Embeddings for Learning Hierarchical Representations
- A Wrapped Normal Distribution on Hyperbolic Space for Gradient-Based Learning