Random-Effects Evaluation of Adaptive Trading Agents

Learned trading and bidding agents are usually judged by a single headline number from a single market. One paper reports a gain in a smart-grid auction, another a revenue figure in one ad exchange on one day, another a return in one simulated stock exchange. Clinical evidence synthesis faced the same situation decades ago and built tools for it. Meta-analysis separates two things that a raw average blurs together. The first is noise within each study. The second is real disagreement between studies, which comes from different populations, different measurement protocols and different outcome definitions. This proposal applies those tools to adaptive trading. Each combination of market, opponent pool and time period is treated as one "study." Each strategy's gain over a common simple reference is treated as an effect size. We then estimate how much of the reported advantage is a stable property of the method, and how much depends on the environment it was measured in.

The core claim has two parts. Between-environment variance (written $\tau^2$, the spread of true effects across studies) dominates the gains reported for learned trading agents. As a result, the honest forecast for a new, unseen market is a prediction interval, and for several methods that interval will include "no better than a simple heuristic." The second part concerns where that variance comes from. Much of it should be explained by how the evaluation was done, specifically whether it used a static historical backtest or a simulator in which the agent's own orders move prices. This is the trading analogue of the finding by Zhang et al. that sarcopenic-obesity mortality effects depend sharply on the diagnostic criterion used. If both parts hold, the field has a principled way to say what a new trading agent will do somewhere it has not been tested. If they fail, it has evidence that current single-environment claims transfer better than feared.

Why this intersection, and what is not already done

The adaptive-trading literature supplies the methods and the motivation. Chandlekar et al. train a bidder with deep deterministic policy gradient (DDPG, an actor-critic method for continuous actions) for PowerTAC double auctions. It learns a continuous valuation scale factor and beats the VidyutVanika broker by 21.42% and the Zero Intelligence baseline by nearly 46% in cost efficiency. Jin et al. use multi-agent reinforcement learning (RL) on one Taobao ad-exchange train/test day pair and reach 18,199 CNY of revenue against 16,359 for DDPG and 5,347 for manual bidding. Ma et al. have large language models write executable trading tools and test them in a simulated exchange where prices respond to orders. They report 13.95% total return with zero drawdown there, alongside NASDAQ, CSI and crypto tests. Ma et al. also flag look-ahead leakage and missing market impact in static backtests as an open problem. Hu et al. show that fixed linear bid shading fails against sellers who adapt. Each paper motivates the proposal in the same way: its gains are measured under a protocol, metric and baseline of its own choosing. That is exactly the outcome-definition heterogeneity that, in clinical work, forces reviewers to give up on quantitative pooling (Qin).

The study-heterogeneity literature supplies the tools. These are random-effects pooling, the $I^2$ statistic (the share of observed variance due to real between-study differences) and moderator stratification, as used by Zhang et al., who report $I^2 = 64.3\%$ that subgroups by setting and diagnostic method largely resolve. The literature also supplies a constraint. PyHealth 2.0 reports that conformal prediction intervals (distribution-free intervals calibrated on held-out data) break under shift across hospital systems and eras. Uncertainty intervals that ignore between-site structure therefore cannot be trusted, and our intervals must be tested out of environment rather than assumed valid.

The closest work bridging the two directions points elsewhere. The auction-based federated learning survey puts RL bidders into data markets and treats heterogeneity only as non-IID client skew. The Karma traffic paper conditions policies on agent attributes through hypernetworks, which are networks that generate another network's weights. Jin et al. cluster merchants to tame agent heterogeneity. In all three, heterogeneity lives among the agents inside one environment. None of them treats heterogeneity across evaluation environments as the object of study.

We considered and rejected the more obvious combination suggested by the research file: RL agents bidding for clinical trial datasets priced by how much they reduce $I^2$. The traded goods there have no strategic sellers, so a greedy selector ranking studies by marginal variance reduction would very likely match an RL bidder. The adaptivity would add nothing.

Core Claim

For an environment $k$ (one market, opponent pool and era), let $y_k$ be a strategy's standardized gain over a fixed reference strategy, with sampling variance $s_k^2$ estimated from seeds and block-bootstrapped episodes. We fit

$$y_k = \mu + u_k + e_k,\qquad u_k \sim N(0,\tau^2),\quad e_k \sim N(0,s_k^2),$$

where $\mu$ is the average true gain across environments, $u_k$ is environment $k$'s true deviation from it, and $e_k$ is within-environment noise. The quantity that matters for deployment is the prediction interval for a new environment,

$$\hat\mu \pm t_{K-2}\sqrt{\hat\tau^2 + \widehat{\mathrm{SE}}(\hat\mu)^2},$$

where $K$ is the number of environments and $t_{K-2}$ is the Student-t quantile. The t form reflects that $\tau^2$ is itself estimated from few environments. We predict the following pattern. Learned agents will show large $\hat\mu$ but even larger $\hat\tau$, so their prediction intervals will be wide. Meta-regression on an "evaluation protocol" moderator (static backtest versus impact-aware simulator) and an "opponent adaptivity" moderator will explain a large share of $\tau^2$. Static backtests will show inflated effects.

Research Program

The first component is a harmonized effect size. Different papers report cost efficiency, revenue, return on investment and total return, each against a different reference such as Zero Intelligence, manual bidding or buy-and-hold. Clinical synthesis meets the same problem with endpoints like life-threatening versus major bleeding (Zhang et al., TAVR). We define the primary effect as the per-episode difference in a strategy's native profit metric from a common, environment-appropriate simple reference, divided by the reference's per-episode standard deviation. This is analogous to a standardized mean difference. As a sensitivity check we use a log ratio for positive-valued metrics. The reference is fixed by rule: a zero-intelligence-style random bidder in auctions and buy-and-hold in asset markets. This prevents authors from choosing a weak baseline per environment.

The second component is the environment pool. It draws on the platforms in the file. PowerTAC double auctions are varied by opponent broker set and simulated season. The Agent-Trading-Arena codebase from Ma et al. provides both its impact-aware simulated exchange and historical NASDAQ, CSI and crypto backtests, which makes the protocol moderator directly testable within one codebase. The SUMO/RouteRL Karma setup from the Karma traffic paper adds a non-monetary bidding market. We will use the Taobao log simulator of Jin et al. if it is obtainable; the file does not say it is public. Splitting each platform by era and opponent pool gives on the order of 20 to 40 "studies." This is comparable to the 23 cohorts in Zhang et al..

The third component is the strategy roster, held to a shared compute and tuning budget. It contains static heuristics (Zero Intelligence, linear shading, MACD-style rules), a DDPG scale-factor bidder in the style of Chandlekar et al., PPO and TD3 agents (two standard deep RL algorithms), and an LLM tool-synthesis agent following Ma et al. where the environment's latency allows it. Each strategy is fit by restricted maximum likelihood (the standard estimator for $\tau^2$) with Hartung-Knapp-adjusted intervals. We then run meta-regression on the moderators: protocol, opponent adaptivity (fixed versus learning opponents, following Hu et al.), number of agents, and market type.

Baselines

The naive reference point is current practice. A strategy is scored by its mean gain in the single environment where its authors evaluated it, with a confidence interval from seeds only. As a concrete anchor, Chandlekar et al. report a 21.42% improvement over VidyutVanika and nearly 46% cost-efficiency improvement over Zero Intelligence on PowerTAC. Jin et al. report 18,199 CNY for their multi-agent bidder against 16,359 CNY for DDPG on Taobao. Ma et al. report 13.95% total return on their simulated exchange. We found no reported result on Lacuna for any between-environment variance or cross-market prediction interval for these methods. Our protocol counts as beating this reference if it predicts held-out-environment effects better, as defined in the decisive checks.

The strongest competing summary method is pooled mean plus or minus standard deviation over all runs in all environments. This is what a careful practitioner would report for multi-environment results. We run it on exactly the same runs. It differs from our model only in not separating within- from between-environment variance, so any difference in coverage isolates that ingredient.

The strongest competitor for calibrated forecasts is split conformal prediction with whole environments as the exchangeable units, where each environment is held out entirely during calibration. It needs no Gaussian assumption. With enough environments it may match or beat random-effects intervals. It is run on the same leave-one-environment-out folds.

Two ablations isolate the key ingredients. A fixed-effect (inverse-variance) pool sets $\tau^2 = 0$ and tests whether modeling between-environment variance matters. Random effects without moderators tests whether protocol and opponent moderators actually explain variance, or whether the gain comes only from widening intervals. Average rank across environments is included as the non-parametric ranking baseline for the question of which strategy is best.

Why not just…?

Why not just report mean and standard deviation across seeds and markets? Doing so merges two variances that answer different questions. Seed noise shrinks with more runs. Between-market variance does not, and only it governs what happens in a new market. The leave-one-environment-out check below measures directly whether the pooled standard-deviation interval undercovers. If pooled intervals turn out to cover as well as ours, that is a real finding: it would mean $\tau^2$ is small and current reporting is adequate. We will report it as such.

Why not just use conformal intervals over environments? This is the most serious threat. PyHealth 2.0 reports conformal failures under cross-site shift, but those failures came from calibrating on samples from the wrong distribution. Environment-level conformal calibration avoids that particular failure. Its weakness is sample size. With 20 to 40 environments, exact conformal intervals are coarse, and they say nothing about which moderators drive the variance. We will compare coverage and width head to head as the number of environments shrinks from 40 to 10. If conformal dominates at every size, we will adopt it for intervals and keep random effects only for moderator attribution.

Why not just test every new agent on more markets? More markets are necessary, but they are not sufficient. Without a harmonized effect size and a variance decomposition, more environments produce a longer table of incomparable numbers, which is the situation Qin describes in clinical ML. The protocol is meant to make more testing cumulative across papers, not to replace it.

Why not just compare against strong heuristics, since learned agents already beat them? The reported margins over heuristics come from single environments. Our hypothesis is that those margins are environment-specific. The meta-analysis will show whether the gap over Zero Intelligence or linear shading survives as a prediction interval excluding zero. If it does, that strengthens the case made by Chandlekar et al. and Hu et al. rather than undermining it.

Decisive Checks

The first check is out-of-environment calibration. In leave-one-environment-out evaluation, the nominal 95% random-effects prediction interval should cover the held-out environment's observed effect close to 95% of the time. The pooled mean-and-standard-deviation interval should cover substantially less, and the fixed-effect interval less still. If random effects does not beat the pooled interval on coverage at equal or modest extra width, the core claim fails.

The second check is the protocol moderator. Within the Agent-Trading-Arena environments, which share code, strategies, and time periods, we compare the same strategy's effect in static historical backtests against its effect in the impact-aware simulated exchange. If the effect is not larger in backtests, the leakage-and-impact concern raised by Ma et al. does not translate into measurable bias, and we report that.

The third check is about decisions. We compare two ways of choosing a strategy: by single-environment mean, or by the lower bound of its prediction interval. Selection by the lower bound should give a better worst-decile realized gain on held-out environments. If it does not, the heterogeneity estimates are statistically sound but practically irrelevant.

We will report $I^2$ alongside $\tau^2$ for every strategy, so readers can compare the share of real disagreement in these trading results with the 64.3% in the clinical pool of Zhang et al..