Pooling gene-expression microarray studies from many hospitals gives more statistical power than any single study. The trouble is that the reliable way to pool them, reprocessing the raw probe files centrally, needs data that institutions rarely release. Ramasamy et al. show that reprocessing raw data avoids the biases of combining published gene lists, and they also report that only about 10% of microarray data is available in raw form. The obvious fix is to have each site send a compact summary that can be combined exactly. Mergeable sketches are built for this. A mergeable sketch is a small data structure in which the summaries of two datasets combine, often by plain addition, into the summary of their union. One example is DDSketch, which counts values in fixed logarithmic buckets so that sketches from different servers add bucket by bucket.
This note argues that applying sketches across the whole pipeline is mostly unnecessary, and in one prominent case it computes the wrong quantity. Most stages of a standard pipeline reduce exactly to adding up ordinary sufficient statistics. Two families of stages do not, and those are the only places where approximate mergeable summaries are worth their error. The first is robust, median-based probe summarization. The second is normalization when the sites do not share an identical probe set. We propose to build a federated pipeline that is exact wherever it can be. It would use sketches only at those two points, and it would be checked end to end against centralized reprocessing on the published 25-dataset cancer meta-analysis.
Core Claim
The tempting idea at this intersection is "mergeable quantile normalization": each site builds a DDSketch of its intensities, the sketches are merged, and the merged quantiles serve as the reference distribution. This idea has two problems.
The first problem is that it is not needed when every site uses the same platform. Quantile normalization aligns each array to the average of the arrays' quantile functions, $\hat{Q}_{mean}(p)=\frac{1}{m}\sum_j \hat{Q}_j(p)$, where $\hat{Q}_j(p)$ is the $p$-th quantile of array $j$ and $m$ is the number of arrays (Statistical Properties of the Quantile Normalization Method). That expression is already a sum. Each site can send the element-wise sum of its sorted intensity vectors together with its array count. Adding these vectors across sites reproduces the centralized reference exactly. The sorted vectors also carry no information about which probe had which value.
The second problem is that merging DDSketches answers a different question. Adding bucket counts gives the quantiles of the pooled mixture of all intensities. Quantile normalization needs the average of the quantile functions, which is a different object. The two disagree exactly when arrays differ in scale or shape, and that is the case normalization is meant to correct.
We therefore claim the following. The quantities that genuinely require approximate mergeable summaries are the order statistics computed per probe across arrays, such as the medians used in median polish, and the per-array quantile functions when platforms differ. Even in those places, sketches should be kept per array or per probe and inverted locally. They should not be pooled.
Research Program
The first part of the work is a stage-by-stage decomposition of a representative pipeline:
- background correction and log transformation;
- quantile normalization;
- probe-to-gene summarization, by either median polish or the Gaussian probe model of RPA;
- batch correction with empirical-Bayes ComBat (pyComBat);
- random-effects inverse-variance meta-analysis.
For each stage we write down what statistic the sites must send. We classify it as exactly additive, additive only within an iterative loop (and count the communication rounds), or intrinsically order-statistic based.
Our working expectation for each stage is as follows:
- Batch correction. ComBat's per-gene, per-batch location and scale estimates follow from per-gene Gram statistics (the sums $X^\top X$, $X^\top y$ and $y^\top y$ for a covariate matrix $X$ and expression vector $y$). These add exactly.
- RPA. RPA's probe affinities $\mu_j$ and noise variances $\tau_j^2$ are averages of residuals across arrays. Given the current array effects, which each site computes locally, they become additive. The cost is one exchange per EM iteration.
- Meta-analysis. Per-gene effect sizes and variances add trivially.
- Median polish. The step that resists is median polish. Each probe's effect is a median of residuals over all arrays at all sites, which is not a sum of anything small.
For that step we build per-probe mergeable median summaries. A DDSketch built on raw intensities is equivalent to a uniform histogram on the log-intensity scale, because logarithmic buckets in raw units are uniform buckets in log units. We therefore treat "DDSketch" and "fixed-width log histogram" as one design. The accuracy parameter $\alpha$ then translates directly into a bin width in log2 units.
The second non-additive case is cross-platform normalization. Here each array is summarized by its own sketch or by a fixed grid of quantile probabilities. Each site inverts its sketches locally, and the sites sum the resulting quantile functions on the common grid. This targets the average quantile function, not the pooled mixture.
The analysis has two goals. The first is to bound how bucket width propagates into the error of the normalized expression values. We will check that propagation empirically against the intensity-dependent variance function estimated by Fan et al.'s nonparametric genewise variance method. Sketch error that is small relative to measurement noise $\sigma(x)$ at every intensity $x$ should leave downstream tests unchanged. The second goal is to measure the gap between the pooled-mixture reference and the average-quantile-function reference under the "varying mixing proportion" failure mode that the quantile normalization paper identifies.
Baselines
Naive reference: local processing plus gene lists. Each simulated site runs the full pipeline on its own arrays and sends only its significant gene list or per-gene p-values. These are combined by Fisher's method. Ramasamy et al. argue this is biased. SNet reports gene-level cross-cohort agreement for t-test-style methods as low as 2.38%. This baseline sets the floor that any federated method must clear.
Strongest competitor: centralized reprocessing. All raw arrays are moved to one server and processed with the same normalization, summarization, ComBat and random-effects steps. This is the gold standard rather than a rival. A federated method "beats" it only in communication and data exposure, and must match it in output. On the 25-dataset cancer meta-analysis, Ramasamy et al. report 168 significantly up-regulated and 168 down-regulated genes at 1% FDR. We use their gene filtering and FDR threshold, and rerun the centralized pipeline ourselves on identical inputs so that both arms see the same data.
Precision reference for "exact". pyComBat agrees with R ComBat to a mean difference of about $-1.06\times10^{-7}$ on an ovarian cancer dataset (pyComBat, via inmoose). We take that as the bar for the stages we claim are exact.
Ablation 1: sketches everywhere. Pooled DDSketches are used for normalization and for every other stage. This isolates the cost of using the wrong target.
Ablation 2: no sketches. Median polish is replaced by additive federated RPA. This isolates whether sketches are needed at all.
Success criteria. Exact stages must match centralized output to floating-point precision. Sketched stages succeed if they reproduce the centralized significant gene sets at 1% FDR with Jaccard overlap above 0.95 while sending at least an order of magnitude fewer bytes than raw files.
Why not just…?
Why not just centralize the raw .CEL files? Where that is possible, it is the right answer, and it is our reference. The obstacle is not computation. It is availability and governance: roughly 10% raw-data deposit, according to Ramasamy et al.. The proposal matters only in the regime where raw arrays cannot leave a site, and it is judged by how closely it reproduces centralized output.
Why not just send summary gene lists or p-values? This is cheap and already practiced. However, the same meta-analysis paper documents cutoff and pipeline biases in this approach, and SNet's reproducibility numbers show how unstable gene-level results are across cohorts. Our naive baseline measures this gap directly on the same data.
Why not just use an additive probe model like RPA and skip sketches entirely? This is the real threat. RPA's statistics are additive given the array effects, and RPA reportedly outperformed RMA in some AffyCompII comparisons. If federated RPA matches centralized results in a few rounds, then sketches are needed only for cross-platform normalization and for sites committed to median polish. Ablation 2 decides this, and we will report that outcome plainly if it occurs.
Why not just merge DDSketches for normalization? As argued above, merged buckets estimate the quantiles of the pooled mixture rather than the average quantile function. Ablation 1 quantifies the resulting distortion. Same-platform sites need no sketch at all, because the sums of their sorted vectors are exact.
Why not use an interactive private protocol like DpBayeSS? DpBayeSS targets user-level local privacy on integer domains. It needs multiple adaptive rounds, and its authors note that continuous domains are hard. Our setting is a handful of institutions with continuous intensities. We do not claim formal privacy here; adding it would be follow-up work.
Proposed Experiments
Experiment 1: the cancer meta-dataset. We simulate multi-center settings by splitting the arrays of each study in the 25-dataset cancer meta-analysis across two to ten virtual sites. We then run all arms. We report three things: the maximum absolute deviation of normalized expression from the centralized pipeline, the Jaccard overlap of up- and down-regulated gene sets at 1% FDR, the bytes sent and the number of communication rounds. The anchor for the gene-set comparison is the published 168/168 counts.
Experiment 2: summarization accuracy on AffyCompII. We compare centralized median polish, sketched median polish with $\alpha$ swept from 0.05 down to 0.001, and federated RPA. Accuracy is measured by recovery of the spike-in concentration gradients. We found no reported numerical AffyCompII score on Lacuna for RPA, so all scores will be generated within this study.
Experiment 3: cross-platform normalization on MAQC Agilent data. The MAQC Agilent data and the ENSAT dataset (13,056 probes by 18 arrays) are used here. We give arrays deliberately different probe subsets and compare three references: the fixed probability grid, per-array sketches with local inversion, and pooled sketches. We overlay the sketch-induced error on the estimated noise function $\sigma(x)$.
Decisive checks. Three outcomes would settle the claims:
- The exact stages match centralized output to roughly $10^{-7}$.
- Sketched median polish at a practical $\alpha$ keeps gene-set overlap above 0.95.
- The pooled-sketch ablation measurably distorts normalization when arrays differ in scale.
If federated RPA alone passes the first two checks, the conclusion is that sketches are needed only for cross-platform alignment. That would still be a useful, publishable map of where mergeability matters.
References
- Key Issues in Conducting a Meta-Analysis of Gene Expression Microarray Datasets
- DDSketch: A fast and fully-mergeable quantile sketch with relative-error guarantees
- Statistical Properties of the Quantile Normalization Method for Density Curve Alignment
- RPA: Probabilistic analysis of probe performance and robust summarization
- pyComBat, a Python tool for batch effects correction in high-throughput molecular data using empirical Bayes methods
- Nonparametric estimation of genewise variance for microarray data
- Finding consistent disease subnetworks across microarray datasets
- Lightweight Protocols for Distributed Private Quantile Estimation