Detecting Emerging Methods from Private Results Tables

Most tools that spot emerging research fronts read text and citations: they count which words are suddenly frequent, or they measure where a paper sits in the citation network. These signals arrive late. A citation network can only rank a paper once other papers have cited it, and a keyword only bursts once many authors have adopted it. Experimental papers carry an earlier signal in their results tables. A new method starts appearing as a comparison row in other groups' tables. Its margin over the best baseline on a shared benchmark grows. New benchmark columns appear. The earliest copies of these tables often sit in places nobody can pool, such as drafts under review, industrial evaluation logs, and institutional reports.

We propose to detect emerging methods by having each institution run a table-reading agent locally. The agent turns its own results tables into normalized records of the form (method, benchmark, metric, value, year). The institutions then send only exact summary statistics to a coordinator. From these, the coordinator fits a growth model and reports each method's excess growth rate with a confidence interval. In two sentences: results tables contain leading indicators of emerging methods that prose and citations miss, but reading them requires reliable table reasoning and, for unpublished tables, a way to combine evidence without moving data. We test whether that combination yields earlier and better-calibrated trend detection than the strongest text and citation baselines, and we treat the claim that tables add lead time as falsifiable.

Core Claim

The proposal makes three claims, each testable on its own. The first is that table-derived signals give earlier detection than abstract or citation signals. These signals are adoption as a baseline in other groups' tables, the velocity of the performance margin, and the appearance of new benchmark columns. The gain should be largest for disruptive methods, which citation embeddings are known to miss. The second is that the bottleneck to using these signals is table extraction, not the trend model. Errors in reading tables propagate directly into false trends, so the extractor must be a program-executing table reasoner rather than a model that reads serialized text. The third is that "is method $m$ emerging?" is a question about a statistical coefficient, so the federated layer must estimate that coefficient exactly. Prediction-oriented federated averaging is the wrong tool.

Each claim is grounded in prior work. For the first, the motivating evidence comes from two studies. Asatani et al. place papers in a citation embedding and derive an "intrinsic publication year" (IPY), a score of how far a paper sits toward the network frontier. IPY tracks bandwagon trends well, but it initially gave the 2009 perovskite solar cell breakthrough a low score because that work had decoupled from established directions. TRENDNERT, a trend-detection benchmark over 160,000 computer science papers, found that full text degraded trend detection badly relative to abstracts (MAP 0.07 versus 0.36). This finding constrains our design. Naively adding full-text tokens hurts, so tables must enter as structured, parsed quantities and not as more text.

For the second claim, the constraint comes from the table-reasoning literature. Serialized tables strip row and column structure (Abdollahzadeh et al.). Arithmetic and comparisons degrade under chain-of-thought prompting (Akhtar et al.). Skipping explicit type cleaning produced the worst accuracy in PoTable's ablations because of silent failures on string-typed numbers. A results table full of "±", bold markers, and footnoted entries is exactly this failure case.

For the third claim, the federated design is constrained by Li et al.. They show on clinical structured data that first-order federated methods (FedAvg and its variants) can predict well but bias coefficient estimates. Second-order exact aggregation (GLORE, Newton-Raphson over summed gradients and Hessians) instead recovers unbiased coefficients with confidence intervals in fewer than 6 rounds.

Research Program

The system has three parts, and we build each from existing methods where possible.

Local extraction agent. This agent follows the plan-then-execute design of PoTable. It loads each table into a dataframe, cleans types explicitly, and runs generated Python with self-debugging. It borrows two pieces from other systems. From TART it takes the separation of planning from computation. From TIDE it takes triple-based verification, rewritten here as (method, outperforms, baseline, on benchmark) triples and checked against the parsed numbers. The agent's job is narrower than general table QA. It identifies which rows are the paper's own method and which are baselines, normalizes method and benchmark names against a shared vocabulary that the coordinator distributes, and emits records. Our contribution at this stage is not a new table reasoner. It is the measurement of how extraction error turns into trend error.

Growth model. For method $m$, institution $i$, and year $t$, let $n_{mit}$ be the number of results tables at $i$ in year $t$ in which $m$ appears as a compared row. We fit a Poisson regression

$$\log \mathbb{E}[n_{mit}] = \alpha_m + \gamma_i + \delta_t + \beta_m\,(t - t_0) + \log E_{it},$$

where $\alpha_m$ is the method's baseline rate, $\gamma_i$ absorbs institution size and style, $\delta_t$ absorbs field-wide growth in publication volume, $E_{it}$ is the number of tables at $i$ in year $t$, and $t_0$ is the start of the window. The coefficient $\beta_m$ is the method's excess growth rate. A method is flagged as emerging when the lower confidence bound on $\beta_m$ exceeds zero.

A second model handles margin velocity. It regresses the method's margin over the best competing row on each shared benchmark against time, with benchmark fixed effects. A third count model tracks benchmark columns appearing. Following the speciation-versus-splitting distinction of Andrei & Arandjelovic, we treat new columns as a candidate "splitting" signal. That paper tracks topic drift and bifurcation in prose with a hierarchical Dirichlet process. Here the parsed table schema supplies concrete labels, which addresses the semantic gap they report in naming their clusters.

Federated layer. We deliberately use the simplest exact option. Each institution sends gradient and Hessian contributions for the models above, protected by secure aggregation and optionally differential-privacy noise, and the coordinator runs Newton steps in the manner of GLORE. We do not claim a new federated optimizer. We claim that exactness is the property that matters for trend inference, and we test this against FedAvg directly. Tabular federated updates are known to leak records (TabLeak; Data Leakage in Tabular Federated Learning), so we also run a reconstruction attack against the shared statistics. The question is whether an institution's unpublished results, such as a specific method-benchmark-value record, can be recovered. Academy supplies a realistic deployment substrate for the local agents. It is middleware for running agents across sites and provides neither table reasoning nor trend models, which is what we add.

Experimental Setting

We simulate closed silos honestly rather than claiming access to private drafts.

The main testbed is the TRENDNERT computer science corpus, which has gold trends defined over 2000–2015 and a 2016 gold set. We partition papers by first-author affiliation into institutions and designate a fraction (10%, 30%, 50%) as closed. The public-data baselines never see closed institutions' tables. The federated system sees them only through aggregated statistics. This isolates the value of evidence that cannot be pooled, separately from the value of table signals.

The disruptive-breakthrough test reuses the solar-cell Web of Science setting from Asatani et al.. We ask whether perovskite efficiency entries in results tables trigger a positive excess-growth flag earlier than IPY ranked perovskite papers highly. The file reports this failure only qualitatively, so the comparison is a lead time in years rather than a comparison against a published number.

We measure extraction accuracy separately. We use a hand-labeled subset of results tables drawn from AnaBench, a benchmark of 63,178 scientific table and figure instances from post-2023 papers. We report record-level precision and recall on method, benchmark, and value. WikiTQ and TabFact accuracies serve only as sanity checks that our reimplementation of the extractor is faithful.

Baselines

The naive reference point is keyword burst detection on abstracts, run centrally on public institutions only. This is the standard bibliometric practice exemplified by CiteSpace-style studies such as the nano-agriculture bibliometric study. It shows what is achievable with no tables, no learning, and no federation.

The strongest text baseline is the best abstract-trained configuration on TRENDNERT. It reports a MAP of 0.36 and recovery of 90% of the 2016 gold trends in TRENDNERT. We rerun it on the same public partition with its published configuration. It receives the same hyperparameter search budget as our growth model, counted in configurations evaluated on a 2010–2012 validation window.

The strongest citation baseline is IPY from Asatani et al.. That paper reports a correlation of 0.419 with future citations on the APS archive, against 0.247 for PageRank. No reported result was found on Lacuna for IPY on TRENDNERT. We compute it ourselves from line-based embeddings of the public citation subgraph, and we map paper scores to method scores by averaging over papers that introduce each method.

Beating these baselines means two things. The first is a higher MAP against TRENDNERT gold trends at a fixed detection year. The second is a positive median lead time, in years, between our first significant flag and the year each gold trend is first detected by the abstract baseline, with a bootstrap interval over trends that excludes zero.

Two ablations isolate the key ingredients. An abstracts-only version of our pipeline, using the same Poisson model and federation, shows whether the table signal itself carries the gain. A serialized-table extractor, which prompts the same language model with Markdown tables and uses no code execution or type cleaning, shows how much extraction quality matters. Two federation conditions bracket the federated layer. Centralized pooling of all institutions' records is the upper bound. FedAvg on the same Poisson model, run to the same wall-clock budget, is the first-order competitor. For the FedAvg comparison we report bias of $\hat\beta_m$ and 95% interval coverage on semi-synthetic data, where we inject methods with known growth rates into the real table records.

Why not just…?

Why not just use abstracts centrally, as TRENDNERT does? This is the most serious threat. Abstracts already yield the best published MAP, and full text made things worse. If the abstracts-only ablation matches the full system in both MAP and lead time, the table thesis fails, and we will report that. Our reason to expect otherwise is mechanistic, not proven. A method's appearance as a baseline row in someone else's table is a commitment by an independent group, and that commitment usually precedes the group writing the method's name into its abstract. The perovskite case gives a concrete test of whether margin velocity moves earlier than prose.

Why not just share differentially private counts and skip federated learning? For the count model this is largely correct. The Poisson model's sufficient statistics are aggregated counts, so exact Newton aggregation and noisy count sharing are nearly the same protocol, and we say so. They differ for the margin model, which has continuous covariates and benchmark effects, and in returning calibrated intervals that include the privacy noise. We run DP count release with post-hoc regression as an explicit baseline. If it matches our coverage and MAP, we adopt it as the federated layer, and the contribution narrows to the table signal and extraction.

Why not just paste tables into a strong language model and ask which methods are emerging? This approach inherits every documented failure of serialized tables. TART reports a 41.9% gain for code-augmented models over chain-of-thought on its table-tool benchmark (TART), and Liu et al. find shortcut pattern-matching instead of table logic. It also cannot produce a calibrated growth estimate or run across silos. The serialized-table ablation measures how much of the trend error this causes.

Why not just run FedAvg, which predicted better than pooling in Li et al.? Li et al. report a q-FedAvg AUC of 0.811 against a pooled AUC of 0.789 on MIMIC-IV-ED. That is a prediction result. The same paper shows that first-order methods distort coefficients, and a trend claim is a claim about a coefficient. The semi-synthetic coverage experiment tests directly whether FedAvg's interval for $\beta_m$ covers the injected growth rate.

Decisive Checks

Three outcomes would settle the proposal.

The first concerns the table signal and the closed silos. If, at 30% closed institutions, the federated table system does not exceed the TRENDNERT abstract baseline's MAP of 0.36 on the same gold set, the table-signal claim is rejected. It is also rejected if the system shows no positive lead time over that baseline.

The second concerns extraction. We plot trend MAP against extraction F1 across three conditions: the serialized extractor, the program-executing extractor, and gold records. If MAP is flat across these, extraction is not the bottleneck and the agent's complexity is unjustified.

The third concerns the federated layer. Exact aggregation should match pooled $\hat\beta_m$ to numerical precision with nominal coverage, while FedAvg should show measurable bias. If FedAvg is unbiased on these simple models, Li et al.'s concern does not transfer to trend inference, and the cheaper method should be used. Alongside these, the reconstruction attack must fail to recover individual unpublished records at the chosen privacy level. Otherwise the system offers institutions no reason to participate.