Do the Scientists Who Cite a Tool Become Its Maintainers?

Most scientific software is written by one graduate student in one lab, published alongside one paper, and then left to decay when the student graduates or the grant ends. Such tools survive only if people outside the original lab start contributing. For the long tail of niche tools, those people usually never find the project. The natural candidates are already visible in data we have: the researchers who cite the tool's paper depend on the tool, yet they typically have no recorded interaction with its code repository. This proposal treats "cited the tool's paper" as a cheap, abundant signal of interest, alongside GitHub stars, forks, and opened issues. It then asks whether that signal predicts the rare, valuable event of someone outside the original lab making a merged code contribution in the following year. The target case is repositories that have almost no activity at all.

The second half of the idea concerns evaluation, not modeling. Every published repository recommender in this space trains and tests on filtered data: active developers, popular repositories, or a handful of mature projects. That filtering removes exactly the neglected tools where sustainability fails. We will build an unfiltered, time-split benchmark of academic repositories and report every result separately for each activity level. We will also measure directly how much the usual filtering inflates reported accuracy.

Core Claim

For academic repositories with fewer than two past outside contributors, a model that includes paper-citation edges will rank the developers who actually contribute in a held-out future window substantially higher than two alternatives: models that use GitHub behaviors alone, and text retrieval that matches a developer's own code to the tool's documentation. The gain should be concentrated where GitHub interactions are sparsest.

The claim is falsifiable at three levels. First, citers may almost never become contributors. That base rate is measurable and worth publishing either way. Second, citation edges may be as uninformative as stars. Third, the whole gain may already come from simple text matching.

Where This Sits Relative to Prior Work

Three systems motivate the problem and supply most of the machinery. Each stops short of this question.

GCNRec runs a two-layer graph convolutional network over a weighted developer–repository graph built from four behaviors (Create, Pull Request, Fork, Star) plus README and code text. It targets general GitHub repositories and treats engagement as the goal. It cannot score repositories that have no behavioral edges.

SoftRec fuses developer–developer, developer–task, and task–task matrices to recommend reviewers and assignees within five mature open-source projects and five enterprise projects. Its fusion weights were tuned by trial and error. The authors also warn that low-intent signals such as browsing can mislead assignment. That warning constrains our design: we must test whether learned weighting beats fixed weighting rather than assume it does.

paper2repo links papers to repositories. It uses "bridge papers," meaning papers that print their GitHub URL, as distant supervision, which here means training labels obtained automatically rather than by hand annotation. We reuse its bridge-paper construction to attach citation graphs to repositories. It solves a static question, "which repo belongs to this paper," and collects repositories through co-starring, which biases it toward popular code. We ask a dynamic, person-level question instead: who will maintain this repo next.

The sustainability literature supplies the target and the motivation. Software Engineering as Instrumentation for the Long Tail of Scientific Software argues that long-tail tools decay partly because other scientists cannot find them and rebuild them instead. It offers an organizational remedy rather than any measurement. Cycling on the Freeway and The Research Software Sustainability Gap describe the pattern of an initial burst of activity followed by stagnation. That pattern tells us to evaluate on repositories whose original authors have gone quiet. Stajich et al. argue that survival requires moving beyond a single lab, which is exactly the event we predict.

The sparse-interaction literature supplies methods and one important caution. Contrastive Meta Learning with Behavior Multiplicity (CML) learns personalized weights that transfer signal from frequent behaviors (views, carts) to rare ones (purchases). SimGCL shows that adding small random noise to embeddings, with a contrastive loss, spreads representations evenly and helps long-tail items more cheaply than dropping graph edges. We use both as ingredients, not as the contribution.

Xu, Zeng and Ai found that on e-commerce, text used alone as the predictor did not beat plain matrix factorization. Text did help as a regularizer on the sparse 0-core splits, where users and items are kept even if they have a single interaction. We use text only as a regularizer, and we explicitly test whether their conclusion transfers to software. The software domain differs because a developer's own code is itself a strong description of their skills.

Finally, Arazo et al. show that under compute budgets, preferentially sampling "important" training examples introduces bias and does not beat uniform sampling. That is a caution against training only on active entities. Their evidence comes from vision tasks, so whether it holds for sparse graphs is itself a question we test.

Research Program

Benchmark. We identify academic repositories as those linked from bridge papers, following paper2repo. We then join their GitHub histories, mined with GHTorrent-style crawls as in GCNRec and the SmartSHARK mining platform, to the citation graph of each bridge paper.

To connect citing authors to GitHub accounts, we require a verified link, such as a GitHub URL printed in the citing author's own papers or an ORCID listed on the profile. We report how much coverage this linking loses, since it bounds what the citation signal can do.

Time split and target. We fix a cutoff date $t$. Everything before $t$ is input. The positive label for a developer–repository pair is a merged pull request or commit to that repository during $(t, t+12\text{ months}]$ by someone outside the original author set.

No k-core filtering is applied (k-core filtering drops users and items with fewer than k interactions). Repositories are grouped by the number of outside contributors they had before $t$: zero, one, 2–5, and more than 5. A separate "at-risk" subset contains repositories whose original authors made no commits in the 12 months before $t$.

Model. The graph contains developers and repositories connected by five behavior types: cite-paper, star, fork, open-issue, and past-contribution. The backbone is a light graph convolution model with SimGCL-style embedding noise. Behavior-specific edge weights are either learned per user, as in CML, or fixed, and this choice is an ablation.

README and code text enter only as a regularizer that pulls a repository's embedding toward its text encoding. This follows the JRL design and lets zero-edge repositories still receive embeddings. Training samples developers uniformly across the whole unfiltered graph.

Scale analysis. We train and evaluate each method on the same graph under two regimes: filtered the way GCNRec and SoftRec filter, and unfiltered. The difference in scores on the low-activity groups directly measures the benchmark inflation that Section 4.4 of the research file hypothesizes.

Baselines

Naive references. The first is ranking developers by overall GitHub contribution volume, a pure popularity baseline. The second is ranking a repository's own citers by how often they cite it. The second is the simplest possible use of our signal, and if it matches the full model, the model is unnecessary.

GitHub-only multi-behavior model. This is our architecture with the cite-paper behavior removed. It is essentially GCNRec's four behaviors with a stronger backbone. For reference, GCNRec reports 69.04% Hit Rate@15 on its Dataset-L against 61.05% for the community-partitioning baseline CPRec, on popular general repositories. Neither number comes from this task, and no reported result was found on Lacuna for future-contributor prediction on academic repositories.

GCNRec as published. This is the two-layer network with its four content features. It is included to connect our numbers to the literature.

BPR-MF over all five behaviors merged into one implicit matrix. BPR-MF is standard matrix factorization trained with a pairwise ranking loss. It is the lightweight model that Xu et al. found hard to beat.

Text retrieval. Each developer is embedded by their own repositories' READMEs and code. Repositories are ranked for them by embedding similarity. This is the strongest competitor for zero-edge repositories.

SoftRec-style joint matrix factorization with fixed weights, tuned on the validation split.

Fairness conditions. All methods share the same time split, the same unfiltered graph, the same text encoder where text is used, the same hyperparameter search budget of 50 validation trials each, and the same GPU-hour cap.

What counts as success. Beating a baseline means a statistically significant gain in Recall@20 and NDCG@20 on the zero- and one-outside-contributor groups, with bootstrap confidence intervals over repositories, across three cutoff dates.

Ablations isolating the key ingredient. We remove the cite-paper behavior. We replace citation edges with random edges of equal density, which checks whether the gain comes from added edges alone. We swap learned per-user weights for fixed weights. We remove the embedding noise.

Why not just…?

Why not just match developers to repositories by text similarity? For repositories with no edges, text retrieval is a genuine threat. Xu et al.'s negative result on text-only prediction was obtained for e-commerce reviews, which describe products, not the skills of the person reading them. The text-retrieval baseline, evaluated group by group, will settle this.

If text retrieval matches the full model on the zero-contributor group, we will report that the citation signal adds nothing beyond topical similarity. The contribution then reduces to the benchmark and the base-rate measurements. The latency figures from Xu et al. (about 0.38 versus about 0.005 seconds per entry) matter less here, because the candidate pool of academic developers is small.

Why not just list a tool's citers and stop there? This is the second naive baseline, and it may be surprisingly strong. The model earns its place only if it ranks citers better than citation count does and also finds non-citers, such as people who starred related tools or who collaborate with citers. The ablation that removes citation edges, together with the citer-only baseline, brackets how much of the gain the graph adds beyond the raw list.

Why not just add citations as one more fixed-weight edge type in LightGCN or BPR-MF, without meta-weighting? SoftRec's experience suggests that weighting matters and is fragile. CML claims learned weights solve this but has only been tested on shopping data. If the fixed-weight ablation is within the confidence interval of the learned version, we will drop learned weighting and recommend the simpler model. We regard that as a likely outcome.

Why not just train on active repositories, where labels are cleaner? Arazo et al. found that biasing the training distribution toward "important" examples did not beat uniform sampling. More to the point, active repositories are not the ones that need help. The filtered-versus-unfiltered comparison measures how far conclusions drawn from active repositories transfer to the long tail.

Why not accept that sustainability is a funding problem that algorithms cannot fix? Partly, we do. This work addresses only the discovery gap that the long-tail instrumentation paper describes. The citer-to-contributor base rate, and the share of at-risk repositories that have any plausible future contributor at all, will show how large that gap is relative to the problems money must solve.

Proposed Experiments

The first experiment is descriptive: for each activity group and for at-risk repositories, what fraction of future outside contributors had previously cited the tool's paper, and what fraction had any GitHub interaction with it? If citers are under 5% of future contributors, the core claim fails early, and we report the finding.

The main experiment compares all baselines and ablations using Recall@20 and NDCG@20, reported for each activity group, over three yearly cutoffs.

The decisive check has two parts. The full model must beat both the GitHub-only model and text retrieval on the zero- and one-outside-contributor groups. The random-edge control must not reproduce the gain.

The scale experiment reruns every method under GCNRec/SoftRec-style filtering. It reports the change in each method's ranking and its absolute scores on the low-activity groups. Methods that look best on filtered data but lose on unfiltered data would support the claim that current benchmarks mislead sustainability-oriented recommendation.

We also report training time relative to LightGCN on the full graph. This checks that SimGCL's reported efficiency (about 2x LightGCN's cost, versus up to 5.7x for SGL, a variant that randomly drops graph edges) holds at our graph size.