Inherited Themes: Phylogenetic Correction for Mixed-Methods Studies of Software Fork Lineages

When a GitHub repository is forked, the copy inherits its parent's code, documentation, and habits, then drifts. A fork of a fork inherits the drift as well. Fork ecosystems therefore form family trees, and any study that counts traits across forks runs into the oldest problem in comparative biology. Close relatives share traits because of common descent, not because those traits are functionally linked, so treating them as independent data points produces confident false correlations. Biologists solve this with phylogenetic comparative methods, which are regressions whose error terms are correlated according to how much evolutionary history two lineages share. Mixed-methods research is the practice of pairing quantitative measurements with qualitative coding of text or interviews. Its studies of software, organizations, and institutions almost never apply that kind of correction, even when the units sampled plainly descend from one another.

This note proposes a mixed-methods study of cryptocurrency fork ecosystems built on the reconstructed fork tree. The quantitative traits are things like modified wallet addresses, added network endpoints, and swapped release binaries. The qualitative traits are themes coded from commit messages, README diffs, and a small set of maintainer interviews. Both kinds of trait are treated as characters evolving on the tree. The work asks two things. First, do qualitatively coded themes show phylogenetic signal, meaning they cluster by lineage the way body size does in frogs? Second, can the tree tell the qualitative arm which forks to examine, by pointing it at sister forks that diverged sharply rather than at a random or convenience sample? If qualitative themes are inherited, then flat mixed-methods analyses overcount them, and the integration step of such studies needs a lineage correction it currently lacks.

Core Claim

The claim has two parts and both can be tested. The first part is that themes produced by inductive qualitative coding behave like heritable traits on fork trees. Their phylogenetic signal can be measured with the same statistics biologists use. Pagel's λ rescales shared history from 0, where lineage is irrelevant, to 1, where traits wander like a random walk along the branches. Blomberg's K compares how much close relatives resemble each other against what such a random walk predicts. Wherever that signal is non-trivial, associations between coded themes and quantitative fork behaviors are inflated under flat analysis. The second part is that phylogenetic independent contrasts can drive the sampling step of a sequential explanatory design. Contrasts are differences between sister lineages, each of which counts as an independent evolutionary event. Choosing forks for close reading by their contrasts should surface explanatory themes that are independent rather than echoes of one ancestral decision.

What would be new is the qualitative arm. Phylogenetic correction for software lineages is a modest extension of known statistics. The contribution is to treat qualitative codes as tree-borne traits and to use the tree as an instrument for qualitative sampling.

Prior Work and Its Role

Two pieces of prior work motivate this directly. What the Fork? Finding and Analyzing Malware in GitHub Forks scanned 68,879 forks of 35 cryptocurrency projects with fuzzy hashing and static malware analyzers and found 26 malicious forks. It treated every fork as a flat, independent item, never used the parent–child structure, and asked no questions about maintainer intent. We reuse its project set and its static indicators but analyze them on the reconstructed tree. Robust Statistical Methods for Empirical Software Engineering showed that software metrics are skewed enough that classical tests miss nominal error rates even at n = 160. It recommended trimmed means, Yuen's test, and Cliff's δ. Those fixes address non-normality, but all of them still assume independent observations. That leaves our gap untouched, and it also constrains us: whatever lineage-corrected model we fit must also be robust to skew.

The comparative methods themselves come from biology. Predictors of genomic diversity within North American squamates and Exploring Brain Size Asymmetry and Its Relationship with Predation Risk Among Chinese Anurans show phylogenetic generalized least squares (PGLS) in routine use, controlling for shared ancestry across 30 and 99 species respectively. Neither has a qualitative track. The squamate study is also a cautionary example we take seriously: cheap fast-changing markers (mitochondrial DNA) failed to track genome-wide diversity. We expect a similar disconnect between surface code diffs and the intent recorded in text. CoMET, from work comparing continuous trait models, supplies maximum-likelihood contrasts and model comparison between Brownian and non-Brownian evolution.

On the qualitative side, Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding framed assisted coding as a ranking task on the CVDQuoding corpus. Its ConvBERT cross-encoders and zero-shot GPT-3.5 reached expert-judged Precision@4 above 0.5. It also found two limits that constrain our design. Neural models did not beat BM25 lexical retrieval at spotting novel codes, and performance depended on whether the coder worked abstractly or concretely. We therefore use code suggestion only to speed up human coders, never to replace them. The mixed-methods exemplars, Why parents consent to their children's participation in genetic research and Machine Learning Security in Industry: A Quantitative Survey, show the value of finding where quantitative and qualitative tracks disagree. The second of these is also an instance of the flat-independence assumption, since it ran ordinal regressions across organizations as if they were unrelated.

Research Program

The first step is to build the trees. For each of the 35 projects named in the fork study, we re-crawl the fork network through the GitHub API. Their data are reported but not available in the file, so this is a fresh crawl. Topology follows the recorded fork parentage. Branch lengths are measured in two ways that we compare: the number of commits a child has that its parent lacks, and a file-level edit distance between parent and child snapshots. The commit count plays the role of time, and the edit distance plays the role of accumulated change. Forks with zero divergent commits collapse onto their parent, because a byte-identical copy is not an independent observation.

The second step is to code traits. The quantitative traits are binary or count indicators from static analysis: a changed hard-coded wallet or pool address, new outbound endpoints, release assets that differ from the build, the malware flags the fork study used, and the size of dependency changes. The qualitative traits come from inductive coding of commit messages, README diffs, and issue text on a stratified sample of forks. Two coders develop the codebook and report inter-rater reliability, with a code-suggestion model ranking candidate codes for them. The likely themes concern monetization, rebranding, abandonment, and security hardening, but the codebook is built from the material rather than fixed in advance. A smaller explanatory phase of roughly 30 maintainer interviews or written questionnaires follows. Which maintainers are approached depends on the sampling arm described below.

The third step is the analysis. For each coded theme and each quantitative trait we estimate phylogenetic signal. For binary codes we use λ under a threshold, or latent-liability, model. We then fit PGLS-style regressions of quantitative behaviors on coded themes, with the error covariance given by the tree. Evolutionary models are compared in CoMET, and we report robust skew-resistant estimates beside the lineage-corrected ones. In the integration step we sort themes into two groups, those that remain associated after lineage correction and those whose association is carried entirely by a few prolific ancestral forks. The second group is a new and checkable category of mixed-methods artifact.

Baselines

The naive reference point is the analysis such a study would ordinarily run. That is flat logistic regression, or a Mann–Whitney comparison, of quantitative fork behaviors against coded themes, treating every fork as independent, on exactly the same forks and codes. Beating it means showing that its Type I error is inflated on the real trees and that some of its significant associations vanish under correction. A merely different p-value does not count.

The strongest statistical competitor is a generalized linear mixed model with a random intercept for root project, and nested intercepts for first-level forks where these are deep enough. This is what a careful empirical software engineer would actually fit. It is the real threat, because fork trees may be shallow and star-like, and in that case a nested random intercept captures almost all the shared history. It gets the same data, the same fixed effects, and the same robust-error treatment. The tree model beats it only if, in simulation on the real topologies, it holds nominal error and better power where the random-intercept model does not, and if on real data it gives a better held-out likelihood on traits with measured λ well above zero.

The strongest existing detection method is the flat static scanning of What the Fork?, which reports 26 malicious forks among 68,879. We do not claim to find more malware. The comparison is whether tree-aware analysis shows those 26 to be clustered in a few lineages or scattered convergently, which the flat scan cannot say. For assisted coding, the reference is the code-suggestion setup of Qualitative Code Suggestion at Precision@4 above 0.5 on CVDQuoding, with BM25 as the baseline for novel codes. No numeric value for BM25 is reported there. We report our own Precision@4 on fork text purely to describe how much the assistant helped.

Two ablations isolate the key ingredient. In the first, the real tree is replaced by a permuted tree that keeps the same branch-length distribution, which tests whether the specific genealogy matters or only the clustering. In the second, the two branch-length definitions are swapped, which tests whether divergence measured in commits or in content is what carries signal. For the sampling claim, the baselines are random selection and stratified purposive selection of forks for interviews, under the same interview budget.

Why not just…?

Why not just fit a mixed-effects model with a random effect per root project? This may be enough, and the proposal is designed to find out instead of assuming otherwise. If most forks attach directly to the root, the tree covariance is close to block-diagonal and the two models agree. We will report tree depth and the share of covariance that lies off the root blocks for all 35 projects before making any claim. The tree model can earn its place only where the trees have real depth, meaning forks of forks with unequal divergence. If that fails, the qualitative contribution, which is the signal estimates for themes, still stands, while the statistical-correction contribution reduces to "use a random intercept." We would state that plainly.

Why not just use robust non-parametric tests as Kitchenham et al. recommend? Those tests fix skew, not dependence. Robust Statistical Methods for Empirical Software Engineering does not claim otherwise. Our simulation runs Yuen's test and Cliff's δ on traits evolved along the real fork trees, so their error rates under inheritance are measured directly. We also keep robust estimation inside the lineage-corrected models rather than presenting the two approaches as rivals.

Why not just scan forks and skip the qualitative arm, as the fork-malware study did? Static scanning tells you what changed but not why. The squamate result in Predictors of genomic diversity is a reminder that a cheap surface marker can fail to track the deeper quantity of interest. The test here is concrete. If coded intent themes add no explanatory power for fork behaviors beyond the static indicators once lineage is controlled, the qualitative arm is not worth its cost in this setting, and we report that.

Proposed Experiments

The first decisive check is a calibration study. On each real fork tree we simulate null traits under Brownian, Ornstein–Uhlenbeck, and discrete Markov models. We then measure false-positive rates at nominal 5% for flat regression, robust tests, the project random-intercept model, and tree-based regression. No reported result was found in the file for these error rates on software trees, so this experiment produces the number rather than comparing against one. The second check estimates λ and K for every coded theme and quantitative trait, with confidence intervals from parametric bootstrap. The headline question is whether any qualitative theme has signal reliably above zero. If none does, the core claim is falsified. The third check refits every theme–behavior association under each baseline and lists which ones survive correction. A subset of the surviving and the vanishing associations is then checked against maintainer accounts.

The fourth check tests the sampling claim. Under a fixed budget of interviews, we compare contrast-based selection (sister forks with the largest standardized divergence in a target behavior), random selection, and purposive selection. Two outcomes are measured: the number of distinct explanatory codes per interview, and the rate at which new codes stop appearing. If contrast-based sampling yields no more independent explanations than random sampling, the claim that the tree should guide qualitative sampling is rejected, and the proposal narrows to lineage correction alone. Each check is designed so that a negative result is informative and publishable, which is the right standard for a methods transfer whose value is not yet known.