Automated bidding agents now buy most online ad impressions on behalf of advertisers. They do not try to maximize profit. They maximize conversions subject to a return-on-investment target and a budget. Auction theory has a clean way to guarantee such markets stay efficient, called the no-overbidding assumption: no bidder ever bids more than an item is worth to them. Under that assumption, simple auctions provably keep a constant fraction of the best achievable welfare, and without it, second-price auctions can lose almost everything. Yet Efficiency of the First-Price Auction in the Autobidding World shows that conversion-maximizing autobidders want to overbid on cheap impressions to use up slack in their campaign-level return target, and that this behavior pulls the first-price auction's worst-case efficiency down to about 0.457. Theory therefore says a little overbidding helps each advertiser and hurts the market. Nobody has measured how much it helps or hurts once flexible learned bidders compete with one another.
We propose to make the amount of overbidding an explicit, tunable budget inside the training of learned bidders, and then measure the curve it traces. The training method is group relative policy optimization (GRPO), a reinforcement learning method that scores each sampled behavior against other samples drawn for the same situation instead of against a learned value estimate. It suits auction replay well, because one logged campaign-day can be re-run several times with the same competitors and different bid trajectories. On top of it we use the constrained variant of Constrained Group Relative Policy Optimization, which normalizes reward and constraint cost separately within each group. The bidder is allowed to spend at most a fraction $\varepsilon$ of its budget on impressions it won with a bid above that impression's value. Sweeping $\varepsilon$ from 0 upward, in single-agent replay and then in populations of mutually adapting bidders, turns a theoretical tension into a measured trade-off.
Core Claim
The claim has two parts. The first is a method claim: separately normalized constrained GRPO, trained on repeated replays of the same auction sequence, can hold a learned bidder's overbid-spend fraction at a chosen $\varepsilon$ while matching or beating strong pacing and generative bidders on conversions. The second is the scientific claim, and it is the one we care about. In multi-agent populations, the individual conversion gain from allowing overbidding shrinks or reverses once every bidder is permitted it, while platform liquid welfare falls monotonically in $\varepsilon$. Liquid welfare is the welfare measure that caps each advertiser's value at what they are actually willing and able to pay. If this holds, overbidding is a prisoner's dilemma for learned bidders, and a near-zero $\varepsilon$ is a defensible platform rule. If it fails, for example if conversions rise for everyone with no welfare loss, then the worst-case bounds are too pessimistic for realistic learned populations. Either result is informative.
The $\varepsilon = 0$ endpoint has a precise theoretical anchor. The pacing rule of Autobidders with Budget and ROI Constraints: Efficiency, Regret, and Pacing Dynamics bids value divided by a multiplier that is at least one. That is no-overbidding with respect to return-adjusted value, and it is what lets the authors guarantee half of the optimal liquid welfare. Our proposal asks what happens when we relax that conservatism by a controlled amount, using a policy far more expressive than a single multiplier.
Research Program
Setting and constraint. Each advertiser $i$ has a return-adjusted value $v_{ij}$ for impression $j$. This is the predicted conversion probability times the advertiser's target cost per conversion, so a bid at $v_{ij}$ exactly meets the return target on that impression. Over one campaign episode, the policy outputs a bid multiplier at each decision step from features such as remaining budget, elapsed time, recent prices and realized return. It acts through a Gaussian over the log-multiplier. The reward $R$ is the episode's conversions. We track two costs. The first is the return-target violation $C^{\text{ROI}}$. The second is the overbid-spend fraction
$$C^{\text{ob}} = \frac{\sum_j p_{ij}\,\mathbf{1}[\text{won } j,\ b_{ij} > v_{ij}]}{\sum_j p_{ij}\,\mathbf{1}[\text{won } j]},$$
where $p_{ij}$ is the price paid. The training goal is to maximize expected $R$ subject to $\mathbb{E}[C^{\text{ob}}] \le \varepsilon$ and $\mathbb{E}[C^{\text{ROI}}] \le 0$.
Groups as common-random-number replays. For each episode (one advertiser-day with logged competitor bids), we sample $G$ bid trajectories from the current policy and replay each against the identical competitor stream. Each candidate $g$ gets advantage
$$A_g = \tilde A^{R}_g - \lambda_{\text{ob}}\tilde A^{\text{ob}}_g - \lambda_{\text{ROI}}\tilde A^{\text{ROI}}_g,$$
where each $\tilde A$ is that quantity standardized within the group, and each multiplier is updated by projected dual ascent, for example $\lambda_{\text{ob}} \leftarrow \max(0, \lambda_{\text{ob}} + \eta(\bar C^{\text{ob}} - \varepsilon))$.
The separate normalization is the method we borrow from Constrained Group Relative Policy Optimization. That paper proves that standardizing a pre-combined reward minus penalty lets the higher-variance term dominate, so constraints can be violated even as the penalty weight grows. It demonstrates the fix only on gridworld hazards and driving collisions. Conversion counts vary across trajectories far more than a small overbid fraction does, which is exactly the regime where its pathology should bite. Because competitors are held fixed within a group, differences between candidates reflect the bidder's own choices rather than market noise. We expect this to be where most of the variance reduction over a learned value function comes from.
Degenerate groups. Two failure cases need handling. The first is the vanishing-variance problem in GRPO that DisCO formalizes as gradient weight scaling with $\sqrt{p(1-p)}$. The second is the related case where all $G$ candidates violate, or none do, so the standardized cost advantage is zero. When the within-group cost standard deviation falls below a threshold, we substitute the raw mean-centered cost scaled by a running population standard deviation. This keeps the constraint signal alive on very tight or very slack campaigns. Whether this patch matters is itself an ablation.
Populations. In the multi-agent stage, all bidders in a market are GRPO policies sharing one $\varepsilon$. They are trained by iterated best response: each round, one cohort updates while the others are frozen. We repeat this across $\varepsilon \in \{0, 0.01, 0.05, 0.1, 0.2, \infty\}$, where $\infty$ means unconstrained, and record per-advertiser conversions and market liquid welfare after training stabilizes. We also log whether bids cycle, which addresses the open concern that relative-ranking learners interacting in one market could produce feedback loops.
Baselines
The naive reference point is truthful return-adjusted bidding, $b_{ij} = v_{ij}$ with budget throttling, together with the classical PID controllers benchmarked in BAT: Benchmark for Auto-bidding Task. BAT reports that Mystique has the lowest spending RMSE, TA-PID and ALM lead on clicks, and M-PID leads on cost-per-click adherence. No reported numeric result was found on Lacuna for these controllers, so we will rerun them with each paper's tuning recipe. Beating them on conversions at $\varepsilon = 0$ is the minimum bar.
The strongest principled competitor on the constraint question is Algorithm 2 of Autobidders with Budget and ROI Constraints. It is dual-multiplier pacing that bids value divided by the larger of the budget and return multipliers. Its reported guarantees are exact ex-post constraint satisfaction, $O(T^{7/8})$ regret, and at least 1/2 of optimal liquid welfare. It is our $\varepsilon = 0$ welfare reference. A result counts as beating it if GRPO at $\varepsilon = 0$ gets more conversions with zero overbid spend and liquid welfare no lower.
The strongest competitor on raw performance is BidFlow from PlatformBid, a flow-matching generative bidder distilled to one-step bids. It reports a 95% improvement in conversions over traditional PID controllers in PlatformBid's homogeneous setting. BidFlow imposes no overbidding control. We therefore compare it to unconstrained GRPO, $\varepsilon = \infty$, on conversions and target-cost variance, and report its overbid fraction alongside. Beating it means matching its conversions within noise while also offering a controllable $\varepsilon$.
We also run PPO with a Lagrangian penalty and a learned value function. This is the standard constrained deep RL alternative and, in spirit, the critic-based route taken by SORL, whose TaoBao +3.75% conversion result is on production traffic we cannot access. All learned methods get the same simulator, the same number of environment interactions (counting every replay in a GRPO group), the same policy network, and the same hyperparameter search budget.
Two ablations isolate the key ingredients. Naive scalarized GRPO, which normalizes the combined reward minus $\lambda$ times cost, tests whether separate normalization is what makes $\varepsilon$ controllable. GRPO with independent replays, where competitor streams are resampled for each group member, tests whether the common-random-number grouping is what makes GRPO competitive with a learned value function.
Why not just…?
Why not just clip bids at value? For the strict constraint, clipping or a bounded parameterization such as $b = v\cdot\sigma(\cdot)$ really does enforce no-overbidding, and it is simpler than a dual variable. We take this seriously and adopt it: the bounded parameterization is our $\varepsilon = 0$ arm, and we do not claim to beat it there. Clipping cannot express a budgeted allowance of overbidding. The work of Deng et al. and Colini-Baldeschi et al. shows that selective overbidding is individually valuable for value maximizers. The intermediate values of $\varepsilon$ are therefore where the interesting trade-off lives, and only a constrained objective can reach them.
Why not just use dual pacing, which already has guarantees? Algorithm 2 is a strong and honest threat at $\varepsilon = 0$. If GRPO does not beat it on conversions there, the method claim fails, and we will say so. The welfare-curve study still stands, because we can run it with pacing at $\varepsilon = 0$ and GRPO above. PlatformBid's large gap between BidFlow and PID suggests a single multiplier leaves conversions on the table. Algorithm 2 also has no knob for selective overbidding.
Why not just PPO with a penalty? It is a plausible alternative, and the matched-budget PPO-Lagrangian comparison settles it directly. The grounds for expecting an edge for GRPO are specific. Value estimates must track 30x traffic swings in BAT. Grouped replays against identical competitors give an exact baseline without any learned estimate. If PPO matches GRPO, the scientific claim about $\varepsilon$ survives with PPO as the training method.
Proposed Experiments
Stage 1: calibration with a known answer. We first train two GRPO bidders in the complete-information two-buyer sequential multiunit auction of Ahunbay, Lucier and Vetta at $\varepsilon = 0$. Under no-overbidding, that paper derives a three-phase equilibrium (competitive, competition reduction, monopsony), declining prices, and a price of anarchy of exactly $1-1/e$. The check is whether the learned bidders reproduce the price path and welfare ratio, and whether raising $\varepsilon$ produces the blocking behavior that theory says destroys second-price efficiency.
Stage 2: single-agent replay on BAT. We use both the first-price segment (about 10M auctions, roughly 9,000 campaigns) and the VCG segment (about 1M auctions). The latter is where no-overbidding matters most theoretically. Metrics are conversions or clicks, spend RMSE, return-constraint violation, and realized $C^{\text{ob}}$ against target $\varepsilon$. The decisive check is whether separately normalized GRPO tracks $\varepsilon$ within a small tolerance across campaigns while the naive scalarized ablation overshoots. That overshoot is the auction analogue of the constrained-GRPO pathology.
Stage 3: population study on PlatformBid AuctionNet. We use the 48-advertiser homogeneous and heterogeneous settings. For each $\varepsilon$ we report mean per-advertiser conversions, market liquid welfare, platform revenue, target-cost variance, and bid-level dispersion. The decisive check for the scientific claim is the shape of two curves against $\varepsilon$. The method claim requires individual conversions in the unilateral-deviation setting (one bidder at $\varepsilon$, the rest at 0) to rise with $\varepsilon$. The prisoner's-dilemma claim requires symmetric-population conversions to be flat or falling and welfare to be falling. We also report whether liquid welfare at $\varepsilon = 0$ stays above the half-of-optimum level that Algorithm 2 guarantees, measured against an offline optimal allocation computed from logged values.
References
- Efficiency of the First-Price Auction in the Autobidding World
- Constrained Group Relative Policy Optimization
- Autobidders with Budget and ROI Constraints: Efficiency, Regret, and Pacing Dynamics
- DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
- BAT: Benchmark for Auto-bidding Task
- PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective
- Sustainable Online Reinforcement Learning for Auto-bidding
- Optimal Type-Dependent Liquid Welfare Guarantees for Autobidding Agents with Budgets
- Two-Buyer Sequential Multiunit Auctions with No Overbidding