When several learning agents share one team reward, an agent judges each of its own actions by averaging outcomes over whatever its teammates happen to be doing. Early in training the teammates are mostly exploring, so an agent's best action is usually paired with a teammate's wrong one. That pairing earns a large penalty, and the team settles into a safe but mediocre joint behavior. This failure is called relative overgeneralization. The usual fix is optimism: judge an action by its best observed outcome and ignore bad ones. Optimism has two known costs. It chases rewards that were high once by chance, and it makes agents blind to real hazards. Drug-safety surveillance has lived with the mirror image of this problem for decades. A drug's adverse-event rate is averaged over all the drugs it is co-prescribed with, and for small counts the field learned to shrink noisy ratios and act on a lower confidence bound rather than on the raw ratio. Averaging over co-medications can blame a harmless drug for its partner's toxicity. It can also hide a toxicity that appears only in one specific combination.
We propose one statistic for both settings. It is optimistic across the choices of other actors (teammates, or co-prescribed drugs) and pessimistic about noise from the world. For each action, and for each context of partner behavior, we compute a small-count-shrunk lower confidence bound on its value. We then take the maximum over partner contexts. The maximum escapes the safe-policy trap. The lower bound, borrowed from pharmacovigilance practice, rejects one-off flukes. Hazards that appear under every partner context survive the maximum and are still avoided. The main test is in multi-agent reinforcement learning, where ground truth and reported baselines exist. A second, deliberately modest test asks whether the same statistic uncovers combination-only drug toxicities that marginal disproportionality averages away.
Core Claim
Relative overgeneralization and the two failures of optimism come from mixing two different axes of uncertainty. The first is uncertainty over what other actors do, where one should be optimistic because a better partner choice is reachable. The second is uncertainty from the environment, where one should be skeptical because a lucky draw is not reachable on demand. We claim that a plain count-based construction separates these axes well enough to match distributional risk-disentanglement on its home benchmarks. The construction is a maximum over partner contexts of empirical-Bayes lower bounds. Unlike that approach, it needs no hand-set risk levels, and its pessimism fades on its own as evidence accumulates.
For agent $i$ in state $s$, let $b$ range over a set $B_K(s)$ of $K$ partner joint actions. In tabular games this set covers all partner actions. In deep settings it holds the top-$K$ samples from the partners' current policies. Let $n(s,a,b)$ be the number of visits to that cell. Let $\bar{r}(s,a,b)$ be the observed mean return, $\bar{r}(s,a)$ the mean for action $a$ pooled over all partner actions, and $\sigma(s,a,b)$ the observed spread. The utility used for action selection or as the advantage target is
$$U_i(s,a) = \max_{b \in B_K(s)} \left[ \frac{n\,\bar{r}(s,a,b) + \kappa\,\bar{r}(s,a)}{n+\kappa} - z\,\frac{\sigma(s,a,b)}{\sqrt{n+\kappa}} \right],$$
where $\kappa$ is a prior strength that pulls small cells toward the pooled mean and $z$ sets the confidence level. The shrinkage and lower-bound form mirror the empirical-Bayes treatment of small cell counts in multi-item gamma Poisson shrinkers. In deep agents, $\bar{r}$ and $\sigma$ come from the mean and spread of a small ensemble of centralized critics, and $n$ comes from a pseudo-count.
Relation to Prior Work
The motivating diagnosis comes from the classic analyses. Claus et al. show that independent learners converge to stable but non-optimal equilibria when the optimum is surrounded by miscoordination penalties. Palmer et al. frame this as averaging utility over exploring teammates. Negotiated Reasoning proves the pathology disappears under consistent reasoning about teammates' updated policies. Its solution uses pairwise-kernel particle updates and a strictly nested agent ordering. We give up its guarantees in exchange for a per-cell statistic with no agent ordering.
Four methods apply optimism on a single axis, and our proposal targets their shared weakness. Optimistic Multi-Agent Policy Gradient clips negative advantages to zero or leaky values. Explicitly Coordinated Policy Iteration tracks a peak value alongside an average value and pulls the peak down heuristically when the two disagree. MaxMax Q-learning maximizes over sampled next states. Risk-based optimistic exploration synchronizes upper-quantile sampling across agents. Each is optimistic about the environment as well as about partners. That is why the OptiMAPPO and EXCEL authors themselves list sensitivity to lucky stochastic outcomes as a limitation. Our maximum runs only over partner actions, never over environment samples.
The closest competitor is DRIMA. It separates cooperative risk from environmental risk with two implicit quantile networks and fixed risk levels for each. We make the same separation with counts and shrinkage. Environmental noise shows up as a wide interval inside a cell, and partner dependence shows up as spread across cells. We test whether this cheaper route matches DRIMA and whether it copes better when the right risk level changes during training. The Price of Paranoia constrains the design: fixed optimism is exploited by drifting partners, while fixed pessimism collapses cooperation. This is a reason to prefer pessimism that depends on the evidence. Potentially Optimal Joint Actions Recognition warns that monotonic value factorizations miss non-monotonic optima. For that reason our deep variant uses an unfactorized centralized critic ensemble.
On the drug-safety side, the method-supplying work is the empirical-Bayes disproportionality family and its supervised successor. That successor, SADR, encodes Bradford Hill's causality criteria as 27 features. Constraining evidence comes from two sources. Reps et al. show that unsupervised longitudinal detectors barely beat chance and fail for reactions rarer than about 1 in 1,000 patients. Li et al. point out that spontaneous reports have no exposure denominator. Our statistic therefore cannot claim causal incidence, and we evaluate only ranking against planted ground truth. Several studies treat drug combinations purely by marginal disproportionality or co-occurrence mining: the dual checkpoint-inhibitor cardiac study, high-order interaction mining in FAERS, and graph-network polypharmacy prediction. None of them asks whether marginal averaging hides narrow combination effects the way it hides coordination peaks.
The research file's own suggested bridge is to apply advantage clipping to polypharmacy attribution. We reject that bridge deliberately. Ignoring negative feedback is the wrong default in safety surveillance. Our transfer runs mainly the other way, from pharmacovigilance's small-count skepticism into multi-agent learning.
Research Program
The first stage is tabular. We implement the statistic in independent Q-learners on Climbing, Penalty, the stochastic two-step game, 5-agent matrix games, and Max of Three. We add a family of variants that we construct ourselves. These add zero-mean reward noise of increasing variance to every cell, plant one "lottery" cell with a low mean but rare large payouts, and add a "cliff" action that is penalized regardless of what partners do. Together they give a two-dimensional grid of difficulty: coordination depth by environmental noise. Pure optimism should fail along the noise axis, pure averaging along the coordination axis, and our statistic should hold across both.
The second stage is deep. The advantage in a MAPPO-style learner becomes $U_i(s,a_i)$ minus a state baseline. $U_i$ is computed from a five-member centralized critic ensemble over $K$ partner actions sampled from the partners' current policies. We study $K$, $\kappa$, and $z$, and we add a schedule-free check in which $\kappa$ and $z$ stay fixed while pseudo-counts grow. The analysis asks when the maximum over sampled partner actions finds the coordinated cell at all. If the partners' policies put vanishing mass on the optimal partner action, $B_K$ will miss it. We will measure how often that happens, because it is the method's main theoretical weakness relative to negotiated reasoning.
The third stage is the return path to drug safety. We build a semi-synthetic FAERS benchmark. We keep the real co-prescription structure of reports and plant signals into the real report background, so we control ground truth. The planted signals are of three kinds. The first is combination-only toxicities, where drug $D$ is harmful only alongside drug $X$. The second is bystander drugs, which are harmless but usually co-reported with a toxic partner. The third is confounding by indication, where the event depends on the underlying disease, modeled after the tiotropium–lung-cancer pattern in the tiotropium FAERS study. For each drug–event pair, the statistic is the maximum over co-medication contexts of the shrunk lower-bound reporting ratio. Its companion exoneration test is the same bound computed within contexts where the drug's most frequent partner is absent.
Baselines
In the tabular games, the naive reference is independent Q-learning with expected-value updates, the learner Claus et al. analyzed. The strongest reported tabular competitor is EXCEL. On 5-agent matrix games, Explicitly Coordinated Policy Iteration reports a coordination ratio over 0.97, against 0.54 for hysteretic Q-learning and 0.09 for lenient learning. Beating it means matching 0.97 on the noise-free games while keeping a markedly higher coordination ratio than EXCEL once reward noise and lottery cells are added. EXCEL's peak-value tracker should be most exposed there.
On the stochastic two-step game, DRIMA reports a reward of 6.98 against 0.03 for DMIX. On Max of Three, Negotiated Reasoning reports convergence to about 10 while QMIX and MAPPO stay near 0. These are the numbers to match. All tabular methods get the same number of episodes, the same exploration schedule, and an equal grid-search budget over their own hyperparameters, with at least 20 seeds.
In deep settings, the reference is MAPPO. The optimism baseline is OptiMAPPO, for which the file has no reported number on the games we use. Zhao et al. report only that it found the global optimum on Climbing and Penalty and matched or beat baselines on 13 of 19 MA-MuJoCo tasks. The strongest competitor on the actual question, coordination under environmental hazards, is DRIMA on the SMAC explorative, dilemmatic, and noisy scenarios. Its authors report outperforming eight baselines there, but the file gives no exact win rates. We therefore rerun DRIMA from its released design with a matched environment-step budget and a matched tuning budget covering its two risk levels. For continuous control, the reference is the 8853 test return on HalfCheetah-2x3 reported for negotiated reasoning.
Three ablations isolate the key ingredient. The first removes the lower bound, leaving a pure maximum over partner cells. The second removes the maximum, leaving a pessimistic average. The third replaces count-based shrinkage with a fixed-width interval. The claim is supported only if the full statistic beats all three on the noise-by-coordination grid.
On the drug-safety benchmark, the naive reference is marginal reporting odds ratio and information component. The strong baseline is LASSO logistic regression with main effects and pairwise interaction terms, the regression member of the GePaRD Borda ensemble. The file gives no reported number for any method on planted FAERS interactions. As external calibration, SADR reports an AUC of 0.792–0.940 and a mean average precision of 0.640 on single-drug reactions in THIN. Unsupervised longitudinal methods reached an AUC of 0.546–0.597 in Reps et al.. Beating the strong baseline means higher average precision on planted combination-only signals, without lower precision on planted single-drug signals.
Why not just…?
Why not just tune OptiMAPPO's leaky slope? A small slope trades coordination against hazard avoidance with a single scalar. Clipping negatives cannot tell a cliff from a clumsy teammate, and Zhao et al. acknowledge the fluke problem. We sweep the slope on the full noise-by-coordination grid. If some slope dominates our statistic on every cell of the grid, the proposal fails and we will say so.
Why not just use DRIMA? This is the real threat. DRIMA already separates partner risk from world risk and works on SMAC. Our case rests on two claims. The first is cost: we use counts and a critic ensemble rather than two quantile networks. The second is adaptivity: our pessimism shrinks with evidence, whereas DRIMA's risk levels are fixed hyperparameters, which The Price of Paranoia suggests is fragile under drift. The decisive comparison is SMAC noisy scenarios plus a mid-training shift in reward noise. If DRIMA wins on both, our contribution shrinks to a cheaper approximation, and we would report it that way.
Why not just take a max over a centralized joint critic? Without the lower bound this is our first ablation. It inherits MaxMax-style fluke-chasing whenever rewards are stochastic, and the lottery-cell games are designed to expose exactly this.
Why not just add interaction terms to a penalized regression for drug safety? For common pairs, this may work as well as our statistic. For rare pairs, the penalty pushes interaction coefficients toward zero, which is the same averaging-away we target. The planted-signal benchmark varies pair frequency so that we can locate the crossover. If LASSO wins across all frequencies, the drug-safety half of the proposal is withdrawn.
Decisive Checks
The central experiment is the noise-by-coordination grid of matrix games. The hypothesis is falsified if either the pure-maximum ablation or EXCEL matches the full statistic in the high-noise, deep-coordination corner. In that case the lower bound is doing no work. The deep-RL check is the matched-budget comparison with DRIMA on SMAC dilemmatic and noisy maps, including the mid-training noise shift. The drug-safety check has two parts. Average precision on planted combination-only toxicities must exceed interaction LASSO at low pair frequency. The bystander exoneration test must also demote planted harmless partners without demoting planted confounding-by-indication signals, which are a separate failure mode our statistic does not claim to fix. As a final negative-control check, the statistic must not rank known tiotropium-style indication artifacts higher than marginal reporting odds ratio does.
References
- The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems
- Negative Update Intervals in Deep Multi-Agent Reinforcement Learning
- Negotiated Reasoning: On Provably Addressing Relative Over-Generalization
- Optimistic Multi-Agent Policy Gradient
- Explicitly Coordinated Policy Iteration
- Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning
- Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning
- Disentangling Sources of Risk for Distributional Multi-Agent Reinforcement Learning
- The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
- Potentially Optimal Joint Actions Recognition for Cooperative Multi-Agent Reinforcement Learning
- Supervised Adverse Drug Reaction Signalling Framework Imitating Bradford Hill's Causality Considerations
- Comparison of algorithms that detect drug side effects using electronic healthcare databases
- Adverse Events of COVID-19 Vaccines in the United States: Temporal and Spatial Analysis
- Cardiac adverse events associated with dual immune checkpoint inhibitors: a pharmacovigilance analysis from the FDA adverse event reporting system
- Mining and visualizing high-order directional drug interaction effects using the FAERS database
- Modeling polypharmacy side effects with graph convolutional networks
- Adverse drug events associated with tiotropium: a real-world pharmacovigilance study of FDA adverse event reporting system database
- A discovery and verification approach to pharmacovigilance using electronic healthcare data