Balancing Mixed-Geometry Teachers in the Student's Own Metric

When a small model learns from several larger "teacher" models at once (multi-teacher knowledge distillation), each teacher adds its own loss term. The student then has to decide, at every step, how much to listen to each teacher and how much to listen to the ground-truth labels. Two things go wrong. Some teachers' gradients are much larger than others' and drown them out, which is called dominance. Some teachers push the student in opposite directions, which is called conflict. Fixed loss weights cannot track either problem as training proceeds, and the results are notoriously sensitive to those weights. The difficulty is sharpest when the student is a hyperbolic embedding. A hyperbolic embedding places items in a curved space whose volume grows exponentially with radius, so general concepts sit near the center and specific ones near the edge. Because distances blow up near that edge, gradients measured in ordinary coordinates are huge for the most specific items and tiny for the general ones.

We propose to solve the teacher-balancing problem in the student's own geometry. At each step we compute one gradient for the task loss and one per teacher. We form their small $(M+1)\times(M+1)$ matrix of pairwise inner products using the hyperbolic metric rather than the flat coordinate one. We then solve in closed form for the update that makes every objective decrease at the same rate. The concrete claim is that the flat-coordinate version of this calculation, which is what any off-the-shelf gradient balancer does, lets the leaf entities near the boundary decide how teachers are weighted. Correcting for the metric changes which teachers win, and the change is large enough to move low-dimensional knowledge-graph accuracy. A second claim is that the corrected balancing flattens the hyperparameter sensitivity that current cross-geometry distillation explicitly suffers from.

Core Claim and Why It Is Not Already Done

The closest pieces exist separately. MulDE distills several 64-dimensional hyperbolic knowledge-graph teachers into a 32- or 8-dimensional student. It weights teachers with learned relation-specific scaling and "contrast attention" over candidate scores, and it never examines gradient magnitudes or angles. All of its teachers share one hyperbolic model. Geometry Awakening pairs Euclidean and hyperbolic teachers and routes each node to one of them by thresholding a local tree-likeness score (Gromov δ-hyperbolicity). Its authors concede that the routing threshold and the static loss weights are brittle. Their student is Euclidean, so the metric question never arises. DTO-KD supplies the method we build on. It treats task-versus-distillation as a two-objective problem, uses gradients of log-losses so that scales normalize, and solves a 2×2 inner-product system in closed form under an "equal contribution" rule. It handles exactly one teacher and flat geometry. GOVERN resolves multi-teacher disagreement by a majority vote on the sign of each teacher's suggested score change. That is a discrete, one-dimensional rule with no magnitude balancing. Our proposal differs from each: it uses $M$ teachers of mixed geometry, a continuous closed-form balance, and an inner product taken in the student's curved metric.

We also have to correct an overstatement that circulates at this intersection: that flat inner products "create false conflict" on hyperbolic embeddings. The Poincaré ball is conformal, meaning its metric is a position-dependent rescaling of the flat one, $g_x = \lambda_x^2 I$ with $\lambda_x = 2/(1-\|x\|^2)$. At any single point, therefore, the angle between two gradients is the same in either metric, and per-entity conflict is unchanged. What does change is the aggregate. The balancing matrix sums over all entities in a batch. The Riemannian inner product of two Riemannian gradients built from flat gradients $g_i^e, g_j^e$ at entity $e$ is $\lambda_e^{-2}\langle g_i^e, g_j^e\rangle$. The full matrix is therefore

$$G_{ij} \;=\; \sum_{e \in \text{batch}} \lambda_{e}^{-2}\,\langle g_i^{e}, g_j^{e}\rangle \;+\; \langle g_i^{\theta}, g_j^{\theta}\rangle ,$$

where $i,j$ index the task loss and the $M$ teacher losses, and $g^{\theta}$ is the gradient on the flat parameters (relation rotations, curvatures). Leaf entities have large $\lambda_e$ and large flat gradients, so the flat-coordinate matrix is dominated by them. The corrected matrix restores the voice of general, near-origin entities. This makes a sharp, falsifiable prediction: flat and metric-aware balancing should assign different teacher weights mainly on batches that mix hierarchy depths.

Research Program

The method computes, at each step, the log-loss gradients for the task and each teacher. These gradients are restricted to the entity rows touched by the batch plus the small relation parameters. In knowledge-graph embedding these gradients are sparse, and teachers differ only in their distillation head. The $M+1$ gradients therefore come from cheap partial backward passes. They do not require full passes through a deep network, which defuses the cost objection DTO-KD raises against multiple-gradient descent.

We generalize DTO-KD's equal-contribution rule. We seek a direction $d=\sum_i w_i g_i$ whose metric inner product with every objective's gradient is equal. This amounts to solving $Gw \propto \mathbf{1}$, with a small ridge term for stability. Teachers whose weight would come out negative are clipped to zero for that step. This is a continuous analogue of GOVERN's outlier rejection, and it lets us log which teacher was dropped when. We also test a per-relation variant that solves one small system per relation group within the batch. That variant is the gradient-based counterpart of MulDE's relation-specific scaling.

The student is a low-dimensional (8 and 32) hyperbolic rotation model in the style of ATTH, with trainable curvature. The teacher pool deliberately mixes geometries. It contains 64-dimensional hyperbolic rotation and reflection teachers like MulDE's, plus Euclidean rotation-style teachers trained in the same codebase. The Euclidean teachers matter because flat, non-hierarchical relations are better served by flat models, as Geometry Awakening argues for graphs. Distillation targets are teachers' scores over the student's top-K candidates, following MulDE's query mechanism. This keeps the comparison to MulDE clean. A secondary arm adds intermediate geodesic alignment, using Lorentz-distance anchors as in Hyper-ICL, but only if the score-level results hold.

The analysis has three parts. First, we measure how much the flat and metric-aware weight vectors diverge, binned by the depth (norm) of the entities in the batch. Second, we record which teacher receives weight on which relation type, for example hypernym chains versus symmetric relations. Third, we map a sensitivity surface: the spread of final accuracy across a grid of the hyperparameters left over (learning rate, temperature, initial curvature, ridge term). We compare that spread against the spread that static-weight baselines show across their loss-weight grid.

Baselines

The naive reference point is the same 8- and 32-dimensional hyperbolic student trained on labels alone, with ATTH's recipe. This tells us how much any distillation helps.

The strongest existing method is MulDE, run with the identical teacher pool where its design allows. It is the published multi-teacher system for low-dimensional hyperbolic knowledge-graph students, and its relation-specific scaling is a learned answer to the same weighting question. The file reports MulDE-RotH at 32 dimensions reaching Hits@10 = 0.574 on WN18RR (MulDE). Beating it means exceeding 0.574 Hits@10 and its MRR at 32 dimensions on WN18RR under our rerun. Our rerun is the reference if it fails to reproduce the reported figure. For FB15k-237 and YAGO3-10 there is no reported low-dimensional MulDE result found on Lacuna, so our rerun is the only reference there. For 8 dimensions the file reports only that MulDE improves MRR by over 90% relative to standard training, with no absolute value.

Mean-ensemble distillation, which averages teacher scores, is included because GOVERN shows it losing to gradient-aware selection (74.63% versus 76.43% on its QA benchmark). We want to know whether that gap reappears for knowledge graphs. A Geometry Awakening-style router is adapted to entities: it uses each entity's local δ-hyperbolicity in the training graph, with a threshold tuned on validation.

The ablation that isolates our key ingredient is the identical $(M+1)$-objective equal-contribution solver run with flat coordinate inner products. This is precisely what one gets by plugging an existing balancer into a hyperbolic student. A second ablation collapses all teachers into one averaged teacher and applies DTO-KD's 2×2 solver unchanged.

Fairness is enforced by fixing the teacher checkpoints, data splits, embedding dimension, and epoch budget across all methods. Each method gets the same number of tuning trials, with its own weights or thresholds counted as tunable. All methods are reported over five seeds. Wall-clock time per epoch is reported, so that the cost of the extra partial gradients is visible.

Why not just…?

Why not just average the teachers and apply DTO-KD as published? This is the most serious threat. If teachers mostly agree, an averaged teacher already captures their consensus, and DTO-KD's 2×2 balance may suffice. The 2×2 ablation measures exactly this. We expect it to lose mainly on FB15k-237 and YAGO3-10, where hyperbolic and Euclidean teachers should disagree by relation type and averaging blurs that disagreement. If it matches us everywhere, the multi-teacher part of the claim fails and we will say so.

Why not just run a grid search over static teacher weights? DTO-KD reports that static weights cannot follow gradient-norm drift during training. It reaches 79.7% Top-1 on ImageNet with dynamic balancing, 5.2 points over its baseline (DTO-KD). However, a grid over $M$ teacher weights is cheap for knowledge-graph models. We therefore give the grid the same trial budget, and we report both peak accuracy and the spread across the grid. A tuned static grid matching our peak would leave only the sensitivity claim standing.

Why not just use an existing balancer such as MGDA or gradient projection in ordinary coordinates? For the analytic reason above, these weight leaf entities by $\lambda_e^2$ relative to the curved metric. The flat-coordinate ablation is this alternative with every other detail held fixed. If its weights correlate almost perfectly with ours, the metric correction is cosmetic and the proposal reduces to "DTO-KD for $M$ teachers."

Why not just use larger Euclidean embeddings? At 32 dimensions ATTH reports a 16–24% relative MRR gain over Euclidean baselines on WN18RR (ATTH). The proposal targets the low-dimensional regime by design. We nevertheless include a 512-dimensional Euclidean student as a ceiling reference.

Proposed Experiments and Decisive Checks

The main experiments train 8- and 32-dimensional hyperbolic students on WN18RR, FB15k-237 and YAGO3-10. We report MRR and Hits@1/10 with filtered ranking, comparing against every baseline above. A smaller replication runs on Cora, Citeseer and Pubmed node classification with Geometry Awakening's Euclidean and hyperbolic teachers. There the student is made hyperbolic, which directly tests the setting that paper left unexplored.

The first decisive check is mechanistic. On WN18RR, the divergence between flat and metric-aware teacher weights should grow with the spread of entity norms in a batch. If it does not, the core claim is wrong.

The second check is whether the metric-aware solver beats the flat-coordinate solver by more than seed noise, with paired comparisons across five seeds, at 8 dimensions. That is where boundary crowding is most severe and where we expect the largest effect.

The third check is sensitivity. Across the residual hyperparameter grid, the interquartile range of final MRR should be markedly narrower for our method than the range MulDE and the router show across their own weight and threshold grids.

The fourth check is interpretability. The logged weights should favor hyperbolic teachers on hypernym-type relations and Euclidean teachers on symmetric ones. Absent that pattern, any accuracy gain is more likely regularization than principled balancing.