Crediting Pioneer Labs for the Experiments They Steer

When labs pool data to discover something together, such as a material with a target band gap or the parameters of an oscillator, the first lab to share takes most of the risk. It exposes unpublished data, it can be scooped, and its data is used to decide which experiments everyone runs next. A rational lab therefore waits for others to go first, and the collaboration stalls before it starts. Current fixes do not handle this. Funding mandates produce data that is "open in appearance only": requests for data stated as available on request go unanswered or are declined 93% of the time Delivering on NIH data sharing requirements. The machine-learning mechanisms that reward early sharing value data as a fixed pile. They ignore the fact that early data changes which later data gets collected.

This note proposes to value each lab by the experiments its data caused the group to run, not only by the rows it contributed. Concretely, a lab's credit is the drop in the group's final discovery performance when the whole collaborative experimental campaign is replayed from the lab's arrival onward without it. In the replay, everyone else's experiments are re-chosen by the same coordinating rule. This is the counterfactual-baseline idea from multi-agent reinforcement learning, applied to labs choosing when to join instead of agents choosing actions. The credit is paid out as a head start on the pooled model, which is the currency academics actually care about, and no lab is ever left worse off than working alone. The proposal tests whether this credit makes early arrival the best move for each lab without breaking the guarantee that joining beats staying out.

Core Claim

The closest existing work, Incentivizing Time-Aware Fairness in Data Sharing, shows how to make rewards fall with arrival time. It sums Shapley values (each party's average marginal contribution over all orderings of the parties) across arrival intervals, or it discounts higher-order interaction terms by a time-decay factor. Its experiments, however, pool static, pre-existing datasets with simulated arrival times. In active scientific discovery the data is not static. FAIR coordinates which experiment each lab runs next using a Gaussian-process model of the unknown function and an acquisition rule that maximizes Nash social welfare, the product of per-lab gains. Under such a rule, a pioneer's early observations shape every later query.

Our claim is that static coalitional credit, even when made time-aware, misprices pioneers in exactly this regime. The error is largest when labs' observation windows are mismatched with their targets, because there early data does the most steering. A trajectory-counterfactual credit corrects the error and produces earlier equilibrium arrival. Let $\tau$ be the realized campaign, $t_i$ lab $i$'s arrival step, and $G(\cdot)$ the collective information gain at the horizon. Let $\tau^{-i}_{\geq t_i}$ be the campaign replayed with lab $i$'s data and queries removed from step $t_i$ on, with the acquisition rule re-choosing the remaining labs' queries. The credit is

$$\phi_i = \mathbb{E}\left[G(\tau)\right] - \mathbb{E}\left[G(\tau^{-i}_{\geq t_i})\right],$$

where the expectation is over observation noise in the replay. Because the surrogate is a Gaussian process, each replay is cheap. This sidesteps the deep-model retraining cost that Counterfactual Explanation of the Shapley Value in Data Coalitions identifies as the obstacle to counterfactual data valuation.

Research Program

The work has three parts.

Arrival game. We first build a staggered-arrival version of FAIR's benchmark tasks: material band-gap prediction, drug–amino-acid affinity, and Van der Pol parameter recovery. Each lab chooses an arrival step and bears a private exposure cost that falls once its own results are published. This makes the hold-out problem of Incentives for Early Arrival concrete inside an active-learning loop.

Credit and payout. Second, we compute $\phi_i$ for every lab and convert it into a release schedule for the pooled Gaussian-process posterior. Every lab receives at least its standalone posterior immediately, which is the individual-rationality floor. Higher-credit labs see the pooled posterior earlier and at higher fidelity, with fidelity controlled by likelihood tempering as in Chen et al.. The embargo idea follows How to Incentivize Data-Driven Collaboration Among Competing Parties, which uses the order of output delivery as an incentive. That work assumes data is submitted upfront. Here the delivery order is earned dynamically.

Analysis. Third, we check two properties. Is $\phi_i$ non-increasing in $t_i$, the incentives-for-early-arrival property? Zhao proves this property for "reward the first critical player" in threshold games, but nothing guarantees it for our credit. Does a lab that improves its data quality ever lose credit? On Blame Attribution for Accountable Multi-Agent Sequential Decision Making proves that Shapley-style attribution can have this flaw in sequential settings. Our credit is a pivotality-style marginal, so we expect fewer violations, but we will measure them rather than assume them away.

Early-arrival monotonicity could fail, for example when an early lab's data is later made redundant. In that case we fall back to a monotone envelope: a lab's credit at $t$ is capped by its replayed credit had it arrived at any earlier step. This is computable in simulation. In deployment it is not, because earlier counterfactual arrivals did not happen. We state this deployment-time limitation openly as an open problem.

Baselines

Naive reference points. The first reference is FAIR run with simultaneous arrival at step zero. It represents the collaboration with no hold-out problem and so serves as an upper reference for collective gain. The second is independent active learning by each lab. On material design, FAIR reports an average information gain of 223.26, compared with 51.71 for researchers working individually FAIR. We will reproduce both numbers before adding staggered arrival.

Strongest existing method. The main competitor is Chen et al.'s time-aware reward cumulation and time-aware data valuation, applied to the datasets each lab ends up contributing in the same campaign. It is the only method in the file designed to make rewards fall with arrival time while keeping individual rationality. We feed it identical campaigns, the same tempering-based payout, and the same Monte Carlo sampling budget, and we tune its decay factor on held-out seeds with the same budget we spend tuning our release schedule. Chen et al. report that late joiners see higher negative log probability, but the file gives no numeric value; no reported result found on Lacuna for exact numbers on their tabular or vision suites, and none exists on FAIR's tasks. Beating this baseline means two things: earlier equilibrium arrival and higher final collective information gain under best-response dynamics, with no more individual-rationality violations.

Other mechanisms. Reward-the-first-critical-player is applied to information-gain thresholds. It gives the strongest guarantee of early arrival but concentrates reward on one lab. A naive arrival-time discount on Shapley value is also included; Chen et al. show it violates individual rationality.

Ablation. The key ablation keeps our payout schedule but replaces $\phi_i$ with static time-aware Shapley credit. This isolates the trajectory counterfactual as the ingredient responsible for any gain.

Why not just…?

Why not just apply time-aware Shapley to the data each lab contributed? This is the real threat. If static and trajectory credit rank labs the same way, our idea reduces to Chen et al. with extra compute. We predict that they agree on Chen's static tabular suite (Friedman, California Housing, Diabetes), where no data steers acquisition, and that they diverge on Van der Pol recovery. FAIR reports that this is the only task where stitching mismatched observation windows was essential. We will measure rank correlation between the two credits as a function of target–support mismatch. If the correlation stays high on every task, we drop the claim.

Why not just reward the first critical lab? It provably induces early arrival in threshold games Incentives for Early Arrival. Zhao notes that in any single run it looks arbitrary. It also violates the Pigou-Dalton balance that FAIR builds in, under which transfers should favor the labs that have gained least. We will report its arrival times alongside the spread of information gain across labs. If it achieves comparable arrival with tolerable spread, that is a legitimate negative result for our proposal.

Why not just mandate sharing or use federated averaging? Mandates yield "open in appearance only" outcomes, such as only 3% of COVID-19 trials reporting results Watson et al.. Uniform averaging lets a party contribute only 25% of its data and still meet its accuracy goal 94% of the time on EMNIST, against 45% under multiplicative-weight reweighting One for One, or All for All. Neither approach addresses when labs join.

Proposed Experiments

On FAIR's three tasks we run 3 to 10 labs, with private exposure costs drawn from a range in which simultaneous joining is not an equilibrium under uniform credit. We simulate iterated best-response choice of arrival steps under each mechanism and report five quantities: mean equilibrium arrival step, final collective information gain, the cross-lab standard deviation of information gain (FAIR's fairness metric), the fraction of labs below their standalone gain, and the rate of early-arrival monotonicity violations, measured by sweeping each lab's $t_i$ in replay. We also measure performance-monotonicity violations by improving one lab's noise level and checking whether its credit falls.

Two checks are decisive. The first: trajectory credit must reach equilibrium arrival measurably earlier than the static time-aware ablation on the Van der Pol and material tasks, and must match it on Chen's static tabular suite. The second: individual-rationality violations must stay at zero under the release-schedule payout. A single replay costs one Gaussian-process refit per remaining step, so we also report wall-clock time against Monte Carlo time-aware Shapley. Chen et al. note that their method scales linearly in the number of distinct joining times.