In many graphs, such as citation networks, a rare class is not just rare. Its members are usually surrounded by majority-class neighbors. Graph neural networks (GNNs) build each node's representation by averaging in its neighbors' representations. On such a graph they drag minority nodes toward the majority and then report high confidence in the wrong label. This matters most for self-training, where a model labels unlabeled nodes and retrains on its own guesses. In that setting the GNN's confidence is least trustworthy exactly where it is most needed. A large language model (LLM) reading a node's own text does not suffer this pull. But LLMs are not stable either: repeated queries on the same input return different answers, even at temperature zero.
This note proposes to treat that instability as a measurement rather than a nuisance. We query the LLM several times per node. Some queries show only the node's own text. Others show the node's text plus varying subsets of its neighbors' text. Two quantities come out of this. The first is how stable the node's own-text answer is. The second is how much adding neighbor text shifts the answer toward majority labels. That shift is a behavioral trace of the same neighborhood contamination that damages GNNs. We use it to decide which pseudo-labels to trust and which nodes need repair, on text-attributed graphs with long-tailed class distributions.
Core Claim
Our claim is that repeated sampling of an LLM under controlled neighborhood perturbations yields a per-node signal that has two properties. First, it is far less correlated with class frequency than GNN confidence. Second, it identifies minority nodes being absorbed by the majority. Selecting pseudo-labels with this signal should improve minority recall and balanced accuracy over existing LLM–GNN co-labeling. It should also make results more reproducible across reruns, because it replaces single-shot LLM outputs with sampled estimates.
Concretely, for node $v$ with text $x_v$, we draw $N$ answers from the LLM under condition $c$. We then compute two scores.
$$s(v) = \max_y \hat p_{\text{self}}(y \mid v), \qquad a(v) = \hat p_{\text{nbr}}(\mathcal{M} \mid v) - \hat p_{\text{self}}(\mathcal{M} \mid v)$$
The pieces are defined as follows:
- $\hat p_{\text{self}}(y \mid v)$ is the fraction of the $N$ own-text samples that answer label $y$. The stability $s(v)$ is therefore the share held by the most common answer.
- $\hat p_{\text{nbr}}$ is the same fraction pooled over samples that also include randomly drawn neighbor subsets.
- $\mathcal{M}$ is the set of majority classes, estimated from the labeled training nodes.
A node with high $s(v)$ whose own-text mode is a minority class, together with large positive $a(v)$, is our operational definition of an "assimilated" node.
Positioning Against Prior Work
The closest existing system is GNN-as-Judge. It selects unlabeled nodes by GNN influence and splits them into LLM–GNN agreement and disagreement sets. It then preference-tunes the LLM toward the GNN's answer on the disagreement set. It motivates our work, and it is also our main foil. It was evaluated only on balanced few-shot splits. It trusts the GNN as arbiter, and it treats each LLM prediction as a single fixed output. Under imbalance, the GNN is the component that is biased. Deferring to it on disagreements may therefore teach the LLM to suppress minority labels. Our design reverses the direction of trust on minority disagreements, and only where the sampled LLM signal justifies doing so.
UNREAL supplies both the problem diagnosis and the opposite policy. It shows that message passing pulls minority embeddings toward majority centroids. It cross-checks GNN predictions against k-means clusters and discards geometrically ambiguous boundary nodes. Its ambiguity measure lives entirely in the GNN's embedding space. That space is the one that collapses under severe imbalance. Our signal comes from outside it.
Two further papers supply the mechanism we expect the LLM probe to detect:
- GraphSB mines "hard samples", meaning nodes whose features look minority but whose neighbors look majority. It argues that minority–majority centroid distance decays with depth.
- GraphENS shows that GNNs are more sensitive to neighbor changes than to feature changes for minority classes.
Our score $a(v)$ is a text-level analogue of GraphSB's hard-sample criterion. It is computed by an LLM rather than by the GNN being corrected.
On the variability side, three papers constrain the design:
- An Empirical Study of the Non-Determinism of ChatGPT in Code Generation reports that 43.64% of CodeContests tasks give no two equal test outputs across five requests even at temperature 0. Single greedy calls are therefore not a stable reference.
- The Effect of Sampling Temperature on Problem Solving in Large Language Models finds no significant accuracy change between temperatures 0.0 and 1.0 (Kruskal-Wallis $p = 0.403$ on GPT-3.5). This is why we can sample at moderate temperature to expose variability without paying in accuracy. - Stability Analysis of ChatGPT-based Sentiment Analysis documents silent drift in API models. This pushes us to run the main study on locally served open models and to use an API model only as a secondary check.
A survey of large language models for data challenges in graphs lists LLMs as enhancers and generators for imbalanced graphs. It never considers their output variability, which is the gap we occupy.
Research Program
The work has three parts.
Measurement study. On long-tailed versions of text-attributed citation graphs, we compute $s(v)$ and $a(v)$ for a candidate pool of unlabeled nodes. We ask three questions:
- whether GNN confidence is more strongly correlated with the frequency of a node's true class than LLM stability is;
- whether $a(v)$ is predictive of the GNN misclassifying a true-minority node as majority;
- whether $a(v)$ tracks UNREAL's geometric imbalance index and GraphSB's hard-sample flag, or captures different nodes.
This study is valuable even if the downstream method fails, because it tests the premise directly.
Routing pseudo-labels. We then build a pseudo-labeling pipeline in the style of GNN-as-Judge, with one change: the final label and the training objective depend on the scores.
When the LLM and GNN agree, the label is used as in GNN-as-Judge.
When they disagree, the scores decide:
- If the LLM's own-text mode is a minority class and stable, and $a(v)$ is large, we take the LLM label. The node becomes a minority pseudo-label for GNN retraining. It also becomes a target for the structure repair step below.
- If the LLM is unstable, the node is deferred to the GNN, or discarded in UNREAL's manner.
Stability thresholds are set per class on a small validation split. This keeps them from simply reflecting the LLM's own label priors.
Structure repair. For flagged assimilated nodes, we down-weight edges to the neighbors whose inclusion most shifted the LLM toward majority. This targets GraphSB-style repair at the specific offending edges instead of diffusing globally, which GraphSB's own sensitivity to diffusion step count makes risky.
Cost is controlled in two ways. We probe only an influence-selected candidate pool of a few thousand nodes, not whole graphs. We use $N$ around 8 per condition. We report the total number of LLM calls.
Baselines
The naive reference is a GCN trained on the long-tailed labeled set with no correction. As a reported anchor, GraphENS gives 62.72% balanced accuracy for vanilla GCN on Cora-LT, against 72.94% for GraphENS. We will reproduce both on our own splits rather than copy them, since our text-attributed version of Cora may differ. A single greedy LLM call per node on own text, used directly as the classifier, is the naive LLM reference.
The strongest graph-only competitor is UNREAL. The file gives only a relative +10.6% balanced-accuracy gain over the next best baseline at imbalance ratio 100 on Cora. We found no reported absolute value on Lacuna, so we will run the released method ourselves. It gets the same labeled splits and the same number of self-training rounds.
The strongest LLM–GNN competitor is GNN-as-Judge. Its reported 3-shot ogbn-arxiv accuracy is 62.21%, against 50.18% for a zero-shot LLM and 38.33% for GCN. That figure is under balanced shots, and no reported result under long-tailed splits was found on Lacuna. We run it with two judges:
- the plain GNN from the original paper;
- a strengthened judge trained with GraphENS or UNREAL.
The second variant asks whether simply fixing the judge is enough.
Every LLM-based baseline gets the same total LLM call budget as our method. For GNN-as-Judge, the extra calls go into majority voting over samples. The same base LLM, candidate pool size, and hyperparameter search budget are used throughout.
Four ablations isolate the key ingredients:
- routing on $s(v)$ alone, with no neighbor conditions;
- routing on the LLM's token log-probability from one greedy call, in place of sampled stability;
- routing without per-class thresholds;
- skipping the edge repair step.
Beating the strongest competitor means higher balanced accuracy and minority-class recall than the judge-fixed GNN-as-Judge at equal LLM budget, on at least three of four datasets at imbalance ratios 20 and 100, with non-overlapping confidence intervals over five full pipeline reruns.
Why not just…?
Why not just fix the judge by training the GNN with GraphENS or UNREAL and keep GNN-as-Judge unchanged? This is the most serious threat. A rebalanced GNN may remove most of the majority bias that motivates us. We do not argue around it; the judge-fixed variant is a headline baseline. Our bet is that rebalanced GNNs still derive their confidence from contaminated neighborhoods. UNREAL itself notes that its k-means path fails when embeddings have already collapsed. If the fixed judge matches us, the measurement study will still show whether $a(v)$ carries information the fixed judge lacks. If it carries none, we will report that the LLM contributes nothing beyond what rebalancing already provides.
Why not just use one greedy call and its token log-probability as the uncertainty? Log-probabilities are cheaper and may already contain most of the signal on own text. They cannot measure $a(v)$ without separate neighbor-conditioned calls anyway. They are also unavailable from many APIs. The Ouyang et al. result, that greedy outputs vary across calls, suggests a single call's confidence is itself noisy. The log-probability ablation settles whether sampling earns its cost. If it does not, the method should be simplified to log-probabilities plus neighbor-conditioned calls.
Why not just set temperature to zero and treat the LLM as deterministic? Shuyin Ouyang et al. show that this does not remove variability, and it discards the information our method uses. Renze's plateau result means sampling at moderate temperature should not cost accuracy. We verify that on our tasks by reporting greedy versus sampled mode accuracy per class.
Why not just re-weight the loss or use SMOTE-style oversampling? GraphENS and TopoImb already show that these fall short on graphs. TopoImb reports ImbNode macro-F1 of 82.1 against 78.1 for the vanilla GNN (TopoImb). Neither method adds new correctly labeled minority nodes, which is what our pipeline supplies. We still include a re-weighted GCN so that the gain from pseudo-labels is separated from the gain from loss balancing.
Proposed Experiments
Data. We use text-attributed Cora, CiteSeer, PubMed and ogbn-arxiv. Long-tailed training splits follow the GraphENS and UNREAL protocols at imbalance ratios 10, 20 and 100.
Models. The main study uses one open instruction-tuned model served locally. One API model is used for the drift check.
Metrics. We report balanced accuracy, macro-F1 and per-class recall for the rarest third of classes. We also report pseudo-label precision split by true class frequency. Finally, we report the standard deviation of each metric across five complete pipeline reruns with fresh LLM samples. That last metric tests the claim that sampled routing makes imbalanced results more reproducible.
Decisive checks. The premise holds only if three things are true:
- In the measurement study, the correlation between true-class frequency and GNN confidence clearly exceeds that for LLM stability.
- $a(v)$ predicts GNN minority-to-majority errors better than UNREAL's geometric index, by AUC.
- Precision of minority pseudo-labels under our routing exceeds GNN-as-Judge's, at equal call budget.
If the first check fails, the idea is wrong and we stop there. If the first two pass but the fixed-judge baseline matches end-to-end accuracy, the contribution narrows to a diagnostic for assimilation, and we will present it as that.
References
- GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedback
- Geometric Imbalance in Semi-Supervised Node Classification
- GraphSB: Boosting Imbalanced Node Classification on Graphs through Structural Balance
- GraphENS: Neighbor-Aware Ego Network Synthesis for Class-Imbalanced Node Classification
- An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
- The Effect of Sampling Temperature on Problem Solving in Large Language Models
- Stability Analysis of ChatGPT-Based Sentiment Analysis in AI Quality Assurance
- A survey of large language models for data challenges in graphs
- TopoImb: Toward Topology-level Imbalance in Learning from Graphs