Auditable Gaming Resistance for Citation Metrics

Citation metrics such as citation counts, the $h$-index and PageRank-style scores decide hiring and funding. They can also be gamed: authors cite themselves, groups of authors agree to cite each other, and people split one paper's record into several to raise their $h$-index. The defenses built so far are heuristics. Examples include discounting citations between co-authors and propagating prestige through the citation graph. Nobody can say how much a determined group can still gain against them. This proposal borrows an idea from mechanism design: break "cannot be manipulated" into several simple, independent conditions, as Mennle and Seuken did for school-choice-style lotteries. Each condition should be checkable one citation at a time. The goal is a guaranteed upper bound on how much any coalition can raise its score per citation it plants, plus a metric chosen to make that bound small while still tracking real influence.

The core claim is that manipulability of a broad class of citation metrics splits into three local axioms. For metrics that satisfy two of them, the worst-case gain of any cartel is the sum of per-paper effects. That sum can be computed exactly in polynomial time. For recursive metrics like PageRank, the gain is amplified by a feedback factor we can bound. Today's evidence is a lower bound: an attack someone happened to find. This work would turn it into a certificate, an upper bound no attack can exceed, and would expose a design trade-off that current metrics resolve without noticing.

Background and the gap

Strategyproofness means that telling the truth is always a best strategy, whatever others do. In deterministic and randomized voting it collides with efficiency. Proving the Incompatibility of Efficiency and Strategyproofness via SMT Solving proves such an impossibility with a constraint solver. The productive response has been to break the property apart. Partial strategyproofness: Relaxing strategyproofness for the random assignment problem shows that, for random assignment, full strategyproofness equals three conditions:

  • Swap monotonicity: promoting an item never lowers your chance of getting it.
  • Upper invariance: reshuffling items below $A$ leaves your chances for $A$ and anything above it unchanged.
  • Lower invariance: reshuffling items above $B$ leaves the relative chances among items below $B$ unchanged.

Dropping the third condition yields "partial strategyproofness," which comes with a numeric degree $r$ and a clear behavioral reading. The authors note that computing $r$ exactly is exponential. Our proposal supplies the method for the citation setting and addresses that open problem there. Related relaxations constrain which guarantees are attainable. Weak Strategyproofness in Randomized Social Choice shows that permissive results collapse once ties are allowed, and Strategyproof Randomized Social Choice for Restricted Sets of Utility Functions restricts which utility functions agents may have. Both warn that no single relaxation fits every setting.

On the citation side, three lines of work motivate the gap without closing it.

Network metrics. These include The Pagerank-Index and COIRank from Identifying Anomalous Citations for Objective Evaluation of Scholarly Article Impact. COIRank down-weights co-author and same-institution citations and then runs PageRank and HITS. Its authors report that 80.88% of 26,366 Physical Review C articles receive co-author citations. These metrics claim resistance to gaming but prove none.

Content-aware weighting. GraLap, from All Fingers are not Equal, predicts how strongly a paper depends on each reference and reaches 0.90 Pearson correlation with human labels on the ACL Anthology Network. Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics weights references by how well they explain the citing paper's content. Both invite the obvious counter-move of rewriting citation sentences.

Complexity-theoretic defenses. How Hard Is It to Impact the Impact of Your Paper? and h-Index Manipulation by Undoing Merges prove worst-case hardness for some manipulations. The same paper shows that splitting records under Google Scholar's unionCite convention is linear-time, with a median $h$-index gain of 4 points for senior researchers.

The two closest boundary works each stop short.

  • Reputation Systems: An Axiomatic Approach gives axioms for graph ranking and an impossibility theorem. It never models agents choosing their own edges.
  • On Strategyproof Conference Peer Review achieves strict strategyproofness by cutting the conflict graph into disconnected groups. On ICLR-17 this required pruning 100 high-degree authors. That is impossible for an open citation graph, where everyone both cites and is cited.

Our proposal differs from both in one way. The citing author's reference list is treated as a strategic report, and the question is how far the metric's response to that report can be pushed.

Core formulation

A metric assigns paper $j$ the score $s_j = \sum_i w_{ij}(G)$, where $w_{ij}$ is the credit citing paper $i$ sends to reference $j$ in citation graph $G$. A coalition $C$, meaning a cartel or one author's papers, controls the outgoing citations and record merges of its own papers. The honest graph is $G$. A manipulation is an edit set $E$, and the coalition's gain is

$$\Delta_C(E) = \sum_{j \in C} \big[s_j(G \oplus E) - s_j(G)\big], \qquad \rho_k = \max_{C,\,|E| \le k} \frac{\Delta_C(E)}{k},$$

so $\rho_k$ is the worst-case gain per planted edit, the citation analogue of the degree of strategyproofness.

The three axioms translate the Mennle–Seuken triple:

  • Credit monotonicity (the analogue of swap monotonicity): raising a reference in a citer's weighting never lowers that reference's credit. Any metric that rewards citations at all needs this.
  • Self-feedback invariance (the analogue of upper invariance): changing a paper's citations to nodes outside its coalition never changes the coalition's own scores. Counting and fractional credit satisfy it. PageRank violates it, because credit sent out returns through cycles.
  • Bystander invariance (the analogue of lower invariance): changing a citer's weights on some references leaves the relative credit among its other references unchanged. Plain counting satisfies it. Any metric that conserves one unit of credit per citing paper violates it, which echoes the observation that harmonic author-credit rules break lower invariance.

The expected theorem has two parts.

First part. Under credit monotonicity and self-feedback invariance, $\Delta_C$ is additive across the coalition's citing papers. Worst-case gain then decomposes into per-paper terms, and each term is the paper's maximum possible emission of credit. Certification becomes a single pass over the graph instead of a search over edit sets. The argument is that self-feedback invariance leaves only direct edges from coalition papers into the coalition. Their effects add up, because each edge touches only its citer's outgoing credit.

Second part. For PageRank with damping $\alpha$, the gain is at most the additive term times $1/(1-\alpha)$. This form comes from summing the geometric series of random-walk returns inside the cartel. We will prove or tighten that constant.

The trade-off surfaces through the third axiom.

  • Unit-emission metrics cap what a padded reference list can send, but they reward concentration. A cartel paper that drops $n-m$ honest external references multiplies each remaining citation's value by $n/m$.
  • Counting metrics avoid concentration but have no emission cap.

We will prove the relaxed guarantee obtained by dropping bystander invariance. Under a condition analogous to Mennle and Seuken's bound on how indifferent agents may be, namely a floor $L$ on honest reference-list length, concentration gain is bounded by roughly $n/L$. We will also study a capped-emission rule that sends credit $\min(n, L)/L$ split evenly, which interpolates between the two regimes.

Research Program

Theory. The first stage proves the decomposition. It shows the three axioms are logically independent by exhibiting metrics that satisfy each pair. It also proves the additivity and feedback bounds and the concentration bound. Record splitting is covered by adding a fourth local axiom, split invariance: splitting a record into versions does not raise its score. We then check which of sumCite, unionCite and fusionCite satisfy it.

Classification. The second stage classifies existing metrics by which axioms they satisfy and computes certified $\rho_k$ for each: raw counts, fractional counting, the PageRank-index, COIRank, and the intensity-weighted $h$-index (hif-index). COIRank's discounting helps only for coalitions its co-author and affiliation filters can see, so its certificate splits into "known-tie" and "unknown-tie" coalitions.

Design. The third stage chooses, within the family satisfying credit monotonicity and self-feedback invariance, the capped-emission metric that maximizes agreement with human influence judgments subject to a target certified $\rho_k$. It traces the full trade-off curve between gaming resistance and how well the metric agrees with human judgments.

Baselines

The do-nothing reference is raw citation counts and the $h$-index under unionCite, as Google Scholar computes them. Both are known to be gameable in linear time. h-Index Manipulation by Undoing Merges reports a median gain of 4 $h$-index points for senior researchers from splitting just one or two articles. Any proposed metric must certify a lower per-edit gain than this reference while not losing agreement with human judgments.

The strongest existing anti-gaming competitors are COIRank and the GraLap hif-index. COIRank is the only published metric that explicitly targets collusive citations on a large real graph. The hif-index is the best-validated content signal, with 0.90 Pearson correlation for intensity prediction on the ACL Anthology Network. For COIRank the file reports only that it "significantly outperforms" PageRank and HITS on recommendation intensity; no reported result found on Lacuna for COIRank's resistance to planted manipulation. For the hif-index, the file reports better separation of ACL Fellows than the $h$-index but no numeric value. Both will be run on identical graphs with their published parameters. Each also gets the same hyperparameter search budget as our capped-emission rule, tuned on a held-out split for agreement with human judgments.

The PageRank-index is the natural competitor for the claim that global prestige propagation already starves cartels.

Two ablations isolate the key ingredients. The first keeps capped emission but reintroduces PageRank-style recursion, which breaks self-feedback invariance. This measures what the feedback term costs. The second removes the cap, restoring plain counting with bystander invariance. This measures the padding versus concentration trade.

Beating a baseline means one of two things. Our metric either certifies a strictly smaller $\rho_k$ at equal agreement with human judgments, or achieves higher agreement at equal certified $\rho_k$. In both cases, the best attack found empirically must stay below our certificate.

Why not just…?

Why not just red-team each metric with greedy attacks and rank them by damage? This is the most serious alternative, and we run it. Greedy search, the title-similarity splitting heuristic of van Bevern et al., and random planted cartels give lower bounds on manipulability. A metric can look safe only because the attacker was weak. How Hard Is It to Impact the Impact of Your Paper? leaves open exactly how far heuristic attacks get. The decisive comparison is the gap between our certified $\rho_k$ and the best attack found. If red-teaming ranks the metrics identically and the certificates are loose, the theory's practical value shrinks to design guidance, and we will report that. If the certificates are tight, they replace an open-ended attack search with a single pass.

Why not just filter conflict-of-interest edges as COIRank does? Filtering works only on the relationships it can observe. The 70% of suspicious impact-factor spikes that All Fingers are not Equal attributes to low-intensity journal-level citations involve no co-authorship. COIRank also inherits PageRank's feedback loop. Our experiments plant cartels of unaffiliated authors, where the filter should provide no protection. COIRank-style discounts can also be added to our metric, since discounting known edges does not break our axioms.

Why not just partition the graph so no one's evaluation depends on their own citations? Divide-and-Rank in On Strategyproof Conference Peer Review needs disconnected groups and pruning of prolific authors. It also cannot achieve pairwise unanimity, meaning it cannot respect every head-to-head comparison that all evaluators agree on. Self-feedback invariance is the part of its guarantee that survives in a connected graph. Our results show what can be certified once the rest is given up.

Why not rely on computational hardness or on reading citation text? Record splitting under Google Scholar's unionCite convention can be optimized in linear time, which rules out hardness as a shield for the most-used metric. Content weighting adds a manipulable report: the citer's wording. Under credit monotonicity, inflated wording can raise credit only within the paper's emission cap. Our certificate therefore bounds text-injection gains that GraLap alone does not.

Proposed Experiments

Datasets. We use four graphs:

  • the Physical Review C citation graph, about 26–30k articles over 43 years, used by COIRank;
  • the ACL Anthology Network with its 1,640 intensity-labeled paper–reference pairs;
  • a computer-science and physics slice of S2ORC, which has 380.5M resolved citation links in total;
  • the Google Scholar profile sets ai10-2011, ai10-2013 and ijcai-2013 for record-splitting attacks.

Planted attacks. Cartels of size 2 to 50 are planted with $k$ edits per member. The cartels are placed three ways: among co-authors, among same-institution authors, and among unrelated authors matched on field and seniority.

Measurements. For each metric we record certified $\rho_k$, the best attacked $\Delta_C/k$, and rank displacement of cartel members. For agreement with human judgments we use Pearson correlation with the AAN intensity labels, where GraLap's 0.90 is the reference, and separation of ACL Fellows from other authors.

Text injection. A further test injects high-intensity cue words into collusive citation sentences. It measures how far the hif-index rises and checks that the rise stays under the emission cap.

Decisive checks. Three results would decide the proposal. First, the capped-emission metric must be the only one whose best found attack stays within its certificate while its agreement with human judgments stays within a small margin of the hif-index. Second, the recursion ablation must show cartel amplification in line with the predicted $1/(1-\alpha)$ factor. Third, the certificate must be close to the strongest attack we can find. If the gap between certificate and attack is wide on real graphs, the local-to-global claim fails in practice, and we would report it as such.