Who Engages, Not Who Approves: Social Signals as Training-Only Supervision for Rare Rejections in Code Review

Automated code review models learn from the decisions of past human reviewers, and those decisions are lopsided. Most review comments are acknowledgments, questions or nitpicks. Most pull requests (proposed code changes) that pass their build checks eventually get approved. Some of those approvals are also unreliable: a longitudinal study of AI-agent pull requests found that repeat reviewers approve more and inspect less as they gain experience (Habituation at the Gate). A classifier trained on these labels learns to say "looks good." The failure is well documented. The best language models on the Lean 4 Mathlib merge-readiness benchmark reach only 36.0% balanced accuracy, and one coding agent recalls 92.0% of merge-ready changes but only 12.0% of not-ready ones (MathlibPR). This is the same minority-class collapse that left a rumour-stance model at 0.000 F1 on denials (Turing at SemEval-2017 Task 8).

Computational social science offers another way to recover a hidden stance: look at who interacts with an item rather than at what the item says. The audience of a news outlet reveals its ideology (Visualizing Media Bias through Twitter). A pull request's audience behaves in a similar way. Examples include which specialist reviewers get pulled in, whether strict reviewers engage and then go silent, and whether other developers later revert or patch the same lines. We propose to use these audience footprints only at training time, as extra supervision that repairs and enlarges the rare "this change has a problem" class. At test time the model sees only the code change and its discussion thread, and it is trained adversarially so it cannot recognize contributor identity. The social graph therefore supplies labels, not shortcuts, and the code representation has to carry the signal.

Core Claim

Social triangulation in prior work is an input feature. We Can Detect Your Bias averages follower biographies into a vector that the classifier reads at prediction time, lifting Macro F1 on unseen outlets from 48.26 (text with triplet pre-training) to 64.29. Copying that design into code review would teach a model that trusted authors get merged. Such a model fails on new contributors, and it builds reputation bias into an automated gate.

Our claim is that audience behavior is most useful as privileged information: signals available when training labels are built but deliberately withheld from the deployed model. We expect a code-and-thread model trained on triangulated labels, with identity removed adversarially, to raise recall on the rare rejection class on unseen-author splits. We expect it to beat three things: the same model trained on raw labels, class-reweighting tricks, and a model that reads social features directly. We also expect the gain to depend on a structural view of the code. Once identity shortcuts are removed, the remaining evidence has to come from what the change does.

Research Program

The work has three components.

Audience-triangulated labels. For each pull request we build an audience footprint with three parts:

  1. The engagement vector: which reviewers commented, approved or went silent. Each reviewer is summarized by their historical strictness, averaged the way Baly averages follower biographies.
  2. A co-engagement graph embedding, in the spirit of audience overlap (Visualizing Media Bias through Twitter).
  3. Downstream behavior mined from repository history after the decision: reverts, follow-up fix commits by other developers touching the same lines, and reopenings.

A small label model, a regularized logistic combiner, is fit on the subset with trustworthy gold labels. It maps footprints to a soft probability that the change has a substantive problem. That probability is blended with the observed label. Approvals that the footprint strongly contradicts are down-weighted rather than flipped. This is distant supervision in the tradition of stance propagation from seed users (Multilingual Stance Detection in Tweets: The Catalonia Independence Corpus). The known risk is noise: Baly found about 3% of articles contradict their outlet's label. Down-weighting instead of hard relabeling limits the damage.

A code-and-thread student model. The student encodes three inputs:

  1. The diff as text, following CodeReviewer.
  2. A syntax-tree view of the changed region, following the multi-view idea of CODE-MVP and the structure-aware encoder of CLMN.
  3. The review thread, with each reply paired to the comment it answers as in Tree LSTMs with Convolution Units.

A known limitation of structural encoders is that they need compilable code (MMAN requires full Clang analysis). MathlibPR sidesteps this because every snapshot builds, so Lean syntax trees are available. On Apache Java we parse post-change files rather than bare diff hunks.

Identity-adversarial training. Following the gradient-reversal debiasing of We Can Detect Your Bias, an adversary head tries to predict author and reviewer identity from the student's hidden representation $h$, and the encoder is trained to defeat it. The training objective is

$$ \mathcal{L} = \sum_i w_i \, \mathrm{CE}\big(f(h_i), \tilde{y}_i\big) \;-\; \lambda \, \mathrm{CE}\big(g(h_i), a_i\big). $$

The symbols are:

  • $\tilde{y}_i$ is the blended soft label from the triangulation step.
  • $w_i$ is a per-example weight that is lower where observed and triangulated labels disagree.
  • $f$ is the stance or merge-readiness head.
  • $g$ is the adversary predicting identity $a_i$.
  • $\lambda$ sets the strength of the adversarial term.

Pairs that differ in label on similar code are additionally used in a contrastive term, borrowing the opposing-pair penalty of TESTED.

Baselines

Naive references. The first is the majority-class predictor, which fixes the floor for balanced accuracy and minority recall. The second is the reputation prior. This logistic regression uses only author historical acceptance rate, reviewer strictness and repository, computed strictly before each decision. It is the baseline we most fear, because if identity explains most outcomes, the whole code-side apparatus is ornamental. Our model counts as beating it only if it wins on the unseen-author split, where the prior has little to go on. It must also at least match the prior on the seen-author split.

Shallow text classifier. A TF-IDF linear SVM over diff and comment tokens is included because Zotova et al. found shallow models beat deeper ones on skewed stance data (58.77 macro F1 on TW-1O, up from 30.68).

Strongest existing trained model. On MACR we compare against CLMN, which fuses a simplified syntax-tree graph encoder with RoBERTa comment encoding. The file gives only relative findings for CLMN on MACR: removing comments costs 38% F1 and 69% MCC, and cross-project transfer costs 0.3% F1. No absolute F1 for CLMN on MACR was found on Lacuna, so we will reproduce it ourselves. On CodeReviewer's quality estimation task the reference is CodeReviewer's reported F1 of 71.53 (versus 64.16 for CodeT5).

Strongest existing prompted model. On MathlibPR the reference is the paper's best reported result: 36.0% balanced accuracy for the best model under full context, and 12.0% not-merge-ready recall for the Codex agent. We also run Adversarial Review-style structured-disagreement prompting, which reported F1 0.533 on SWE-PRBench, as a prompting baseline designed to resist false consensus.

Fairness conditions. All trained baselines share the student's backbone size, data splits and hyperparameter search budget. Each also gets its own class-weighted and focal-loss variants, so reweighting is never credited only to us. Decision thresholds for every system are tuned on the same validation split.

Ablations isolating the key ingredient. Four ablations are run:

  1. Identical student, raw labels instead of triangulated labels.
  2. Social features as test-time inputs instead of privileged labels.
  3. No adversary.
  4. Flat diff text instead of the syntax-tree view.

"Beating" the field means higher not-merge-ready or rejection recall at matched precision, and higher balanced accuracy and macro F1, on the unseen-author split, with bootstrap intervals that exclude the best baseline.

Why not just…?

Why not just prompt a strong language model with the diff and thread? MathlibPR already ran this experiment with full context and repository access, and the best model reached 36.0% balanced accuracy, with agents collapsing toward approval. Adversarial Review shows that forcing disagreement helps, but it still learns nothing from the community's own record of what goes wrong later. We include both as baselines rather than dismissing them.

Why not just reweight the loss or oversample rejections? Reweighting amplifies the existing minority labels, but our premise is that those labels are scarce and also contaminated by habituated approvals. TESTED's ablations show that how the training set is composed matters more than the loss: removing diversity sampling cost 8 F1 points, while removing the contrastive term cost 3. We still treat reweighting as a serious threat. Every baseline gets it. If raw-label training with focal loss matches triangulated-label training, our label-repair claim fails.

Why not just feed the social signals to the model as features, as Baly did? That will likely win on authors seen during training. Baly's own results also show that sources unseen at training time are where text models suffer (BERT at 35.53 Macro F1). A model that reads reputation is useless or harmful for newcomers and AI agents. Ablation 2 measures exactly this trade-off on seen versus unseen author splits.

Why not just use flat diffs and skip syntax trees? Flat diffs are a strong default, given CodeReviewer's 71.53 F1. Structural views help in retrieval and defect detection (MMAN; CODE-MVP), but no paper in our file shows this for review decisions. Our specific prediction is that the structural view matters more once the adversary removes identity shortcuts. If ablation 4 shows no gap with the adversary on, we will report flat diffs as sufficient.

Proposed Experiments

Datasets.

  • MathlibPR (MathlibPR; 15,895 build-passing snapshots, 4,486 not-merge-ready): trustworthy gold labels for the minority class and guaranteed parseability.
  • MACR (CLMN; over 500,000 accept/reject samples across 11 Apache projects): scale, and per-project transfer.
  • AIDev (Habituation at the Gate; 16,895 human reviews): the habituation stress test.

Every split is temporal. Downstream signals such as reverts and follow-up fixes are mined only from history that is used for label construction and is never visible to the student. Each dataset gets a seen-author split and an unseen-author split, with the latter modeled on Baly's media-based split.

Decisive checks.

  • Imbalance. Does triangulated-label training raise not-merge-ready recall on MathlibPR above the 12.0% agent figure and balanced accuracy above 36.0%? Does it beat the raw-label ablation by a margin outside the bootstrap interval?
  • Shortcut. On unseen authors, does the privileged-information student beat both the reputation prior and the social-features-as-input variant? If the input variant wins even on unseen authors, our privileged-only design is wrong.
  • Habituation. On AIDev, do approvals that our model scores as risky get reverted or patched by others at a higher rate than approvals it scores as safe? Is this enrichment larger for late-experience reviewers, whose labels the habituation study suggests are least reliable?
  • Label-noise audit. We hand-inspect a stratified sample of approvals that triangulation down-weighted, and report what fraction show a substantive defect. This bounds how much of the gain comes from real repair versus regularization.

Positive results on all four checks would show that audience behavior can supervise a code-grounded critic without becoming its crutch. A failure on the shortcut check would be equally informative: it would mean review outcomes are mostly social, and that automated gates trained on them inherit that.