A knowledge graph embedding model stores facts such as (Paris, capital_of, France) by giving every entity a learned vector and scoring each triple with a simple function of those vectors. On graphs with millions of entities, that per-entity table dominates memory. Parameter-efficient methods therefore replace it with a small shared generator: a narrow code that a deep network "lifts" to full width, or a network that builds each entity's vector from its neighbors and relations. Our claim is that these generators quietly give up something that parameter counts do not show. The vectors they produce for all entities may occupy only a thin slice of the embedding space, a failure the self-supervised learning literature calls dimensional collapse. For the bilinear scoring functions most of these models use, the dimensionality of that slice is a hard ceiling on what the model can express.
We propose to measure this directly, and to test whether an explicit penalty borrowed from VICReg can recover the lost capacity at a fixed parameter budget. VICReg is a self-supervised objective that keeps every feature dimension varying and decorrelated from the others. We apply its variance and covariance terms to the generated entity table as an auxiliary loss, and we compute those statistics over entities sampled uniformly rather than over the triples in a training batch. If the idea is right, the accuracy lost when entity storage is compressed should track the effective rank of the generated embeddings better than it tracks parameter count, and restoring that rank should move compressed models up the accuracy-per-parameter curve.
Core Claim
Take DistMult, whose score for $(h,r,t)$ is $\langle e_h, w_r, e_t\rangle$. For relation $r$, the full matrix of scores over all entity pairs is $S_r = E\,\mathrm{diag}(w_r)\,E^\top$, where $E$ is the $|\mathcal{E}|\times d$ matrix of entity vectors and $w_r$ the relation vector. It follows that
$$\mathrm{rank}(S_r) \le \mathrm{rank}(E).$$
The same bound holds for ComplEx and other bilinear scorers. A model with a full table can make $E$ full rank at no extra cost. A generator-based model cannot automatically do so. In LiftNet (From Wide to Deep), every entity starts as a 4- or 16-dimensional code, and only the nonlinearities in the lifting network can push the 512-dimensional outputs beyond a 4- or 16-dimensional linear span. In EARL (Entity-Agnostic Representation Learning), entities with similar relation contexts receive similar inputs to a shared network.
We measure the effective spread using effective rank, $\mathrm{erank}(E)=\exp\big(-\sum_i p_i\log p_i\big)$, where $p_i=\sigma_i/\sum_j\sigma_j$ are the normalized singular values of the centered matrix $E$. Our hypothesis is that under compression, $\mathrm{erank}(E)$ falls well below $d$. We further hypothesize that it explains variation in link-prediction accuracy that parameter count does not, and that the variance and covariance terms of VICReg raise it cheaply.
The file points toward this gap without closing it. Random Entity Quantization showed that topology-aware code assignment over-clusters entities and that random codes work as well or better. That is evidence that distinguishability, and not structure, is the binding constraint. However, it acts only on discrete input codes, and the paper itself says the encoder does the heavy lifting. Neither EARL nor LiftNet reports the covariance or spectrum of the representations it generates.
Research Program
The first stage is diagnostic and has value even if the method fails. For LiftNet (4, 16, 64 base dimensions, with DistMult and ComplEx scorers), EARL, and EARL with random quantization, we will train across a sweep of parameter budgets and record the singular value spectrum and effective rank of the full generated entity table at convergence. We will then relate MRR to effective rank and to parameter count jointly.
To test the causal direction, we will take trained models and truncate $E$ to its top-$k$ singular directions for decreasing $k$. This shows how steeply accuracy depends on rank for each dataset. We also run the reverse: truncating a full-table model to the effective rank that a compressed model reaches, and asking whether the two then score alike.
The second stage adds the regularizer. At each step we draw a batch $B$ of entities uniformly at random, generate their vectors $Z\in\mathbb{R}^{|B|\times d}$ through the shared encoder, and add
$$\mathcal{L}_{\text{aux}} = \mu\,\frac{1}{d}\sum_{j=1}^{d}\max\!\big(0,\gamma-\sqrt{\mathrm{Var}(Z_{:,j})+\epsilon}\big) + \nu\,\frac{1}{d}\sum_{j\ne k}C(Z)_{jk}^2$$
to the model's usual link-prediction loss. Here $\gamma$ is the target standard deviation per dimension and $C(Z)$ is the sample covariance of $Z$. These are exactly VICReg's variance and covariance terms. The ordinary triple-scoring loss plays the role that VICReg's invariance term plays in vision.
Uniform entity sampling is a deliberate choice. Covariance computed over the entities appearing in a batch of triples is weighted by entity degree, so the penalty would decorrelate hub entities while leaving the long tail of rare entities free to collapse. Rare entities are exactly where compressed models are weakest. For EARL we also test an optional invariance term between two views of the same entity, each built with a random half of its connected relations and neighbors dropped. We expect this to matter most in the inductive setting.
The third stage asks where the benefit lives. We will break results down by entity degree and by relation type, and track how effective rank evolves during training. Inference cost is unchanged, because the regularizer touches only training.
Baselines
The naive reference is each compressed model trained exactly as published, with the usual negative sampling and no extra term. We report its effective rank and its MRR at every budget in the sweep. Alongside it we train a full-table model of the same scorer. This anchors the top of the accuracy-per-parameter curve and serves as a control: its $E$ should already be near full rank, so the regularizer should give it little.
The strongest existing method on the parameter-accuracy trade-off in the file is EARL with random quantization. Random Entity Quantization reports an MRR of 0.311 on FB15k-237 with 1.4 million parameters, against 0.310 with 1.8 million for original EARL. It is the most direct competitor because it already targets entity distinguishability, which is the same symptom we attribute to collapse, and it does so with no added loss.
For LiftNet, From Wide to Deep reports 96.9% parameter savings on WN18RR. The file contains no reported MRR for it: no reported result found on Lacuna for LiftNet MRR on FB15k-237 or WN18RR. We will therefore reproduce it ourselves. For the non-sampling reference, Efficient Non-Sampling Knowledge Graph Embedding reports HR@10 of 0.447 for NS-TransE on FB15k-237 with full tables.
All methods share the same code base, data splits, embedding width, number of negatives, epochs, and hyperparameter search budget. The regularizer weights $\mu$ and $\nu$ are counted inside that budget rather than tuned on top of it. Beating EARL with random quantization means either higher filtered MRR at an equal parameter count, or equal MRR at a clearly lower parameter count, across three or more seeds and on at least two of FB15k-237, WN18RR, and CoDEx-L.
The ablations isolate the key ingredients. We compare variance only against variance plus covariance, entity-uniform against triple-batch statistics, and the regularizer applied at LiftNet's narrow base layer against its lifted output. We also run pure VICReg with negatives removed, which is the obvious "negative-free" combination. We expect it to underperform badly, because ranking the true tail among all entities requires the scoring loss to see alternatives, and we report it to rule it out rather than to champion it.
Why not just…?
Why not just add batch normalization or layer normalization to the encoder output? Normalization fixes each dimension's scale but not how dimensions relate to one another. On Feature Decorrelation in Self-Supervised Learning showed that batch normalization prevents complete collapse yet leaves features collinear. In that work, explicit whitening lifted CIFAR-10 accuracy from 86.32% to 89.50%, a vision result that may not transfer. A differentiable whitening layer of that kind is a genuine threat: it attacks the same covariance through architecture instead of a loss. We therefore include batch normalization, layer normalization, and a decorrelated-batch-normalization layer on the generated embeddings as direct competitors under the same budget. If whitening matches the regularizer, we will report that the mechanism is decorrelation and that the loss form is incidental.
Why not just use more negatives, or score against every entity at once? Contrastive pressure does spread embeddings apart, and this may already prevent collapse. If so, the diagnostic stage will show high effective rank in the unregularized models, and that negative result would be worth publishing. The counterpoint is cost. Scoring against all entities requires generating the whole entity table through the encoder at each step, which is precisely the expense compressed models avoid. Efficient Non-Sampling Knowledge Graph Embedding makes all-negative training cheap only for full tables with factorizable square losses. We will sweep the number of negatives up to full 1-vs-all scoring on FB15k-237 and plot effective rank and MRR against training time. The regularizer earns its place only if it reaches comparable rank and accuracy at lower cost.
Why not just use random quantization, or simply a wider base code? Random codes make the inputs distinguishable, but a shared encoder can still map them onto a low-dimensional output. Widening LiftNet's base code raises the rank ceiling directly, but it spends parameters. Both are honest alternatives, and both are handled by comparing everything on the same accuracy-per-parameter axis. If a 16-dimensional base code without regularization beats a regularized 4-dimensional code of equal total size, the regularizer is not worth using at that point on the curve.
Why not just use teacher-student bootstrapping instead? Keeping a moving-average copy of the encoder doubles training memory, which runs against the purpose of the exercise. Halvagal et al. further argue that such schemes amount to an implicit form of variance regularization, which we make explicit.
Proposed Experiments
We use FB15k-237 and WN18RR for the main sweep, CoDEx-L and YAGO3-10 for scale, and UMLS as a small sanity check. We report filtered MRR and Hits@10 against parameter count, the effective rank of the full entity table, and training and inference time. We run the inductive FB15k-237 split for EARL, where the two-view invariance term is expected to matter.
Four checks decide the proposal. First, across budgets and architectures, effective rank must predict MRR beyond parameter count; if it adds nothing, the collapse story is wrong. Second, the gain from regularization must be largest at the smallest budgets and near zero for full-table models. A uniform gain would suggest the penalty is acting as generic regularization rather than restoring capacity. Third, entity-uniform statistics must beat triple-batch statistics on low-degree entities; this is the ingredient that most distinguishes our method from dropping VICReg in unchanged. Fourth, the regularizer must match or beat the whitening layer and the heavy-negative runs at equal training cost.
If the first check fails, meaning compressed encoders do not collapse under negative sampling, we will publish the spectra as evidence that parameter-efficient KGE is limited by something other than geometry, and stop there.
References
- From Wide to Deep: Dimension Lifting Network for Parameter-Efficient Knowledge Graph Embedding
- Entity-Agnostic Representation Learning for Parameter-Efficient Knowledge Graph Embedding
- VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
- Random Entity Quantization for Parameter-Efficient Compositional Knowledge Graph Representation
- Efficient Non-Sampling Knowledge Graph Embedding
- On Feature Decorrelation in Self-Supervised Learning
- Implicit variance regularization in non-contrastive SSL