Type-Centered Energy: Measuring and Preventing Hidden Over-Smoothing in Heterogeneous Graph Autoencoders

Graph autoencoders learn node embeddings without labels. A graph neural network compresses the graph's links and node features into short vectors, and a decoder is trained to rebuild the links from those vectors. When labels are scarce, these embeddings are often all a downstream classifier gets, so whatever the autoencoder destroys is lost for good. One known way to destroy information is over-smoothing: repeated neighbor averaging makes node vectors converge until they can no longer be told apart. The standard way to detect it is Dirichlet energy, which sums squared differences between linked nodes' vectors and falls toward zero as nodes blur together.

We argue that on heterogeneous graphs this measurement is misleading. These are graphs with several node types, such as papers, authors and venues. Linked nodes are usually of different types, and each type's embeddings sit around their own mean. The energy summed over cross-type edges is therefore dominated by the fixed gap between type means. It can stay high while nodes within each type collapse onto one another, which is the collapse that actually ruins few-shot classification. We propose to strip out the type means before measuring energy. We will use this "type-centered energy" first to diagnose where heterogeneous autoencoders lose information, and then as a lower-bound constraint during self-supervised pretraining. The test is whether it improves classification from one to five labels per class.

Core Claim

Heterogeneous graph autoencoders suffer two different collapses that a single energy number conflates. The first is type collapse. An inner-product decoder rewards linked nodes for having aligned vectors, so on author–paper edges it pulls the author and paper type means together. The second is within-type collapse. Deeper or repeated aggregation shrinks the spread of nodes inside each type.

Write $\mu_t$ for the mean embedding of nodes of type $t$, $t(i)$ for the type of node $i$, $\mathcal{E}_r$ for the edges of relation $r$, and $d_i$ for node $i$'s degree under that relation. The type-centered energy of embeddings $Z$ is then

$$ \mathcal{E}_{tc}(Z) = \frac{1}{2}\sum_{r}\sum_{(i,j)\in\mathcal{E}_r} \frac{\big\|(z_i-\mu_{t(i)})-(z_j-\mu_{t(j)})\big\|^2}{\sqrt{d_i d_j}}. $$

Ordinary Dirichlet energy, expanded on cross-type edges, equals $\mathcal{E}_{tc}$ plus a term driven by the distances $\|\mu_{t(i)}-\mu_{t(j)}\|$ plus a cross term. The mean-offset term can keep the total high even when $\mathcal{E}_{tc}\to 0$. Our claim has two parts. First, in existing heterogeneous autoencoders, few-shot downstream accuracy tracks $\mathcal{E}_{tc}$ and within-type variance, not raw energy. Second, enforcing a lower bound $\mathcal{E}_{tc}(Z) \ge c\,\mathcal{E}_{tc}(X^{(0)})$ during reconstruction training buys label efficiency that skip connections and graph-agnostic whitening do not. Here $X^{(0)}$ is the input projected into the shared space and $c$ is a constant.

This builds on Dirichlet Energy Constrained Learning for Deep Graph Neural Networks. That paper proves energy decays exponentially under plain message passing. It keeps energy bounded below through derived residual coefficients, orthogonal weights and a shifted ReLU, which lets supervised networks reach 64 layers. Its analysis uses only the homogeneous Laplacian, it leaves heterogeneous graphs as future work, and it never touches reconstruction objectives. We reuse its lower-bound construction but apply it to a different quantity and a different training signal.

Research Program

We will build three things.

The first is a diagnostic suite. Raw energy, type-centered energy, per-type variance and between-type mean distance will be logged layer by layer and across pretraining epochs. We will apply it to two existing heterogeneous autoencoders. GraMI is a variational autoencoder that reconstructs links plus a low-dimensional hidden feature matrix, which lets it handle missing and noisy attributes. It never examines depth or smoothing. THeGAU is a wrapper whose decoder uses separate MLPs per node-type pair and scores only schema-valid edges. It counters smoothing only with a skip connection from raw features into the final classifier. We expect the inner-product decoder to show type collapse, and THeGAU's type-pair decoder to avoid type collapse while still showing within-type collapse as encoder depth grows. If the diagnostic shows no gap between raw and type-centered energy on real benchmarks, the premise fails, and we would report that.

The second is the constrained autoencoder. We take a GraMI-style encoder with a THeGAU-style type-pair decoder and add two things. One is per-relation residual mixing whose coefficients come from the EGNN lower-bound argument, applied to type-centered features. The other is a hinge penalty that is active only when $\mathcal{E}_{tc}$ falls below its bound. Type means are computed per mini-batch. This keeps the cost linear in the number of sampled edges, and it sidesteps the scaling issue THeGAU reports for full decoding.

The third is the label-scarce evaluation protocol. Frozen embeddings are probed with a linear classifier at 1, 3, 5 and 20 labels per class, using random label selection. A secondary run uses feature-propagation medoid selection from Active Learning for Graph Neural Networks via Node Feature Propagation, to check that conclusions do not depend on which nodes get labeled. We will also run the out-of-distribution few-shot splits of COHF. That paper meta-learns over causally disentangled variational embeddings, and it reports that its backbone must stay at 2–3 layers because deeper versions degrade. We read that as motivating evidence that within-type collapse is already hurting few-shot heterogeneous learning.

Baselines

The naive reference point is a linear probe on the raw type-projected input features, plus a standard inner-product graph autoencoder run on the same heterogeneous graph with all types merged. These show how much any structure-aware pretraining helps at one to five labels. They also give the clearest view of type collapse.

The strongest existing method for self-supervised heterogeneous pretraining is GraMI. It beat HGMAE and HGCA in 30 of 36 test cases across ACM, DBLP and YELP. The file contains no absolute scalar, so we report no reported result found on Lacuna for GraMI's few-shot Macro-F1 on ACM, and we will rerun it ourselves.

The strongest structural fix is THeGAU. It reports a 3.4% Macro-F1 gain on IMDB when wrapped around SimpleHGN. We will run it with its skip-connected classifier, and also with that skip feeding our linear probe, so the comparison is between energy control and a skip connection.

TAGDN avoids deep stacking. It aligns type distributions with closed-form Gaussian optimal transport and then applies one-step diffusion. It reports average gains of 5.02% in node classification and 25.43% in clustering, and stays stable up to 40 propagation steps. It is the strongest competitor that drops the autoencoder entirely.

For the out-of-distribution few-shot splits, the reference is COHF. It reports +5.66% accuracy and +4.32% F1 on inter-dataset transfer over prior heterogeneous few-shot methods.

All methods get the same embedding width, the same pretraining epoch budget and the same number of hyperparameter trials, run by one search procedure. The same label splits are used across 10 seeds. Beating a baseline means a higher mean Macro-F1 at 1 and 5 labels per class, with non-overlapping 95% intervals on at least two of ACM, DBLP and IMDB.

The key ablations remove the energy constraint while keeping everything else fixed. One replaces type-centered energy with raw energy in the constraint, which isolates the centering. Another replaces the graph-aware constraint with per-type whitening of the embeddings.

Why not just…?

Why not just keep the encoder at two layers? This is the most serious threat. GraMI and COHF both run shallow and work. If within-type collapse only appears with depth, a shallow encoder avoids it for free and our constraint is pointless. We will settle it two ways. First, we will check whether type-centered energy and within-type variance already shrink during pretraining at two layers. We expect the reconstruction loss itself to keep pulling linked nodes together, so the collapse would happen over epochs and not only over layers. Second, we will check whether constrained deeper encoders beat the best shallow configuration at 1–5 labels. If the shallow model wins at every label budget, we would narrow the claim to a diagnostic one.

Why not just add residual or initial-feature skip connections? EGNN reports that a plain deep GCN falls from 82.5% to 21.9% on Cora and that heuristic residuals carry no guarantee. EGNN also beat GCNII by 3.1% on ogbn-arxiv at 32 layers. Complete the Missing Half shows that residuals do not restore node identity destroyed by low-pass aggregation, and that adding a high-pass channel gains about 10% on Texas and Cornell. That evidence is homogeneous and supervised, however. The THeGAU-skip baseline at matched parameter count is the direct test in our setting.

Why not just whiten or decorrelate each type's embeddings? Feature decorrelation of the kind used in R²FGC prevents redundant embedding dimensions in homogeneous clustering. Per-type whitening would also stop variance from vanishing. This could plausibly match our method. The difference is that whitening ignores the graph: it can restore spread without keeping linked nodes distinguishable. The whitening ablation decides it. If whitening ties, we will recommend it as the simpler fix and report the energy diagnostic as the contribution.

Why not just use one-step diffusion as TAGDN does? TAGDN's alignment assumes unimodal Gaussian type distributions and clean attributes. GraMI's setting has missing and corrupted attributes, which breaks that assumption. We will run TAGDN on the attribute-masked variants as well as the clean ones, to see whether its advantage survives.

Proposed Experiments

We will use ACM, DBLP and IMDB for the main few-shot probes, and YELP, MovieLens and Douban with COHF's degree and heterogeneity shifts for transfer. ogbn-mag is a scalability check for mini-batch type means. Cora and Texas will confirm that type-centered energy reduces to EGNN's energy when there is a single type.

The first decisive check is correlational. Across all trained models and depths, does type-centered energy predict 5-shot Macro-F1 better than raw energy, measured by rank correlation? If it does not, the core claim is false.

The second check is interventional. On attribute-masked ACM and DBLP, does the constrained autoencoder beat GraMI, THeGAU and per-type whitening at 1 and 5 labels per class under the matched budget described above?

The third check is depth. Does accuracy stay flat from 2 to 16 encoder layers with the constraint, while baselines degrade? If the answer is yes, it would also remove COHF's stated depth limitation when our encoder replaces its backbone.