Ranking-Based Demand Elicitation for Supply Chains, Borrowed from Peer Review

When a manufacturer has too little capacity and must split it among retailers, each retailer has a reason to overstate its demand forecast so that it receives a bigger share. The standard fix is a screening contract, a menu of options priced so that each retailer does best by picking the one that matches its true type. Such contracts are designed one buyer and one product at a time. They become hard to construct once a retailer carries many products and reports pass through intermediate distribution tiers.

Academic peer review faces a similar problem and has a cleaner tool for it. In the isotonic mechanism, authors who submit several papers are asked only to rank their own submissions. The noisy reviewer scores are then adjusted to be the closest scores that respect that ranking. Authors cannot inflate all their papers at once, because a ranking carries no information about overall level, and under suitable conditions an honest ranking is the author's best choice. We propose moving this mechanism, together with the associate-editor layer that runs it, into multi-product capacity allocation. Each retailer ranks its products by private expected demand. An intermediate distribution node, acting like an associate editor, combines that ranking with its own noisy sell-through signals before passing calibrated forecasts to the manufacturer. The central question is whether a coarse, ranking-only report removes the classic "inflate everything" lie while keeping most of the value of private information. A second question is when retailers' profit shapes break the guarantee that honesty is best.

Core Claim

Ranking-based elicitation, run by an intermediate review tier, should recover a large share of the gap between ignoring retailer forecasts and having full honest information. It should do so with no side payments and no per-retailer contract menu, as long as the capacity allocation rule is chosen so that a retailer's profit rises convexly with its estimated demand. When retailer profit is concave in allocation, as it is under newsvendor economics, honest ranking can fail. We will characterize which allocation rules restore it. The novelty is not "use LLM agents with a supervisor." The novelty is a specific truth-telling device from peer review that has not been applied to supply chain forecast sharing, along with its failure conditions.

The mechanism is concrete. Retailer $i$ sells $n$ products with true expected demands $\theta_1,\dots,\theta_n$. The intermediate node observes noisy signals $y_j$, such as Poisson sell-through counts or distributor deliveries. The retailer reports a ranking $\pi$, and the node computes

$$\hat\theta = \arg\min_{x}\ \sum_{j=1}^{n} \ell(y_j, x_j) \quad \text{subject to} \quad x_{\pi(1)} \ge x_{\pi(2)} \ge \dots \ge x_{\pi(n)},$$

where $\ell$ is the negative log-likelihood of the signal model: squared error for Gaussian noise, Poisson deviance for count demand. Capacity is then allocated by a rule $A(\hat\theta)$ that the manufacturer, the executive tier, publishes in advance. This is exactly the structure studied in the Isotonic mechanism for exponential family estimation in machine learning peer review. That work shows truthful ranking for authors whose utility is convex in the adjusted scores. We reuse the estimator and ask what the retailer's utility $U_i(A(\hat\theta))$ looks like under realistic inventory economics.

Relation to Prior Work

Three lines of work motivate the proposal by documenting the gap. Supply Chain Sourcing Under Asymmetric Information shows that informed parties prefer opacity because it protects their bargaining position, so mandated transparency is not a credible default. Promised Lead-Time Contracts Under Asymmetric Information and Mechanism Design for Capacity Planning Under Dynamic Evolutions of Asymmetric Demand Forecasts build screening menus over lead time and capacity reservation. Both are restricted to a single supplier facing a single buyer and price each private parameter separately. Our mechanism has no menu and handles many products jointly. In exchange it elicits less: an order, not levels.

One result constrains the design. The Limits of Optimal Pricing in the Dark shows that richer pricing mechanisms gain nothing against a strategically deceptive buyer, because they overfit to the lie. That finding argues against building ever more expressive elicitation schemes and for coarse reports that limit what a liar can say. A ranking is such a report. We do not claim the ranking defeats Leontief-style lies in general. We claim it removes one specific degree of freedom, uniform inflation, and we measure what remains.

Another set of papers supplies the editorial structure and the agent test bed. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that a designated leader and a critic-coordinator supervisory layer improve coordination among cooperative LLM agents. Its agents have no private incentives, so it says nothing about lying. Agentic LLMs in the Supply Chain and InvAgent supply serial inventory simulators with LLM agents. In both, agents share information in good faith and the central component only routes messages. We keep their simulators and add self-interested reporting. We replace an LLM critic with an estimator whose incentive properties can be proven. Incentive Design in Peer Review: Rating and Repeated Endogenous Matching supplies a second editorial idea we use only in an extension: rating intermediate reviewers across rounds to discourage them from colluding with the parties they review.

The closest crossing in the file is A Blockchain-Based Approach For Embodied Carbon Management Along The Construction Supply Chain. It has intermediate tiers validate supplier data under role-based access, so that reports can be checked without exposing trade secrets. It assumes those tiers behave honestly and contains no incentive mechanism. Our proposal is the incentive layer that paper says is missing, though for demand reporting rather than carbon.

Research Program

The theoretical part characterizes truthfulness. With Gaussian or Poisson signals and the estimator above, we study a retailer whose payoff is newsvendor profit on the allocated capacity. The goal is a condition on the allocation rule $A$ under which honest ranking is optimal. Candidate rules include proportional allocation, allocation linear in $\hat\theta$, and a convexified rule that rewards concentrated capacity. We then bound the efficiency lost by forcing $A$ into this class, relative to the first-best allocation under true $\theta$. A second analysis covers deliberate non-convexity. We compute the retailer's best-response ranking under each rule by enumeration for small $n$ and local search for larger $n$, and report the manipulation gain, defined as best-response profit minus truthful profit.

The structural part asks what the associate-editor tier contributes beyond the estimator. With several retailers per region, the regional node can calibrate across retailers using shared products, which acts like several reviewers scoring one paper. It also adds one period of reporting delay. We measure this trade-off directly. This is the latency-versus-oversight tension the file raises, citing An Analysis of the Tsingtao Beer Merger, and here it becomes a measurable cost-of-delay curve.

The behavioral part replaces optimized liars with LLM retailers prompted to maximize their own profit, using the InvAgent and Moros et al. simulators. This tests whether realistic, imperfectly strategic agents behave more like truthful rankers or more like naive inflators under each mechanism.

Baselines

The naive reference point takes retailer forecast levels at face value and allocates on them. Against strategic retailers this should be clearly worst, and its gap to honest reporting defines the size of the problem. A second floor ignores retailer reports and allocates using only the node's noisy signals $y$. This is the same estimator without the ranking constraint, and it is the ablation that isolates our key ingredient. If ranking does not beat it, the private information is not being used. The upper reference is honest full information, where true $\theta$ is reported. A further ablation feeds the estimator an oracle-honest ranking, which separates the value of the ranking's content from the problem of eliciting it truthfully.

The strongest existing competitor is a per-product screening contract in the style of Oh and Özer's capacity reservation menu, applied product by product. It is the only method in the file with incentive guarantees under asymmetric demand forecasts. Its reported result is an average supplier profit improvement of 80.75%, with a 7.52% average optimality gap against full information, in Mechanism Design for Capacity Planning Under Dynamic Evolutions of Asymmetric Demand Forecasts. We run it on the same demand traces, with its menu parameters optimized under the same simulation budget we give to tuning our allocation rule. Beating it means matching or exceeding its supplier profit at a smaller or equal optimality gap, without side payments, on instances with five or more products.

For the agent experiments, the reference is the information-sharing setup of Moros et al. Its reported result is a 92.8% cost reduction over standalone agents for Gemini Flash in Agentic LLMs in the Supply Chain, obtained with good-faith agents. We rerun it with self-interested retailers, using the same models, prompts other than the profit objective, and demand process. Our claim is that its gain collapses under self-interest while the ranking mechanism keeps most of it. For the multi-product SupplyGraph setting there is no reported result on Lacuna for any forecasting or allocation method, so all numbers there will be produced fresh under the protocol above. The file reports no quantitative results for the isotonic peer-review mechanism either.

Why not just…?

Why not just share all information, as Moros et al. do? Their 92.8% gain assumes agents report honestly. Supply Chain Sourcing Under Asymmetric Information and the carbon-ledger paper both show that informed firms resist or distort disclosure when it costs them leverage or trade secrets. The self-interested rerun of their framework tests this directly. If LLM retailers turn out not to inflate even when told to maximize profit, that is an informative negative result, and the case for our mechanism shrinks to the rational-liar setting.

Why not just penalize forecast errors after the fact with a proper scoring rule? This is the most serious threat. Scoring rules are the textbook way to elicit honest forecasts, and they elicit levels rather than only order. Our argument is that demand under scarce capacity is censored: a stocked-out product hides its true demand. An after-the-fact penalty therefore cannot distinguish a retailer that inflated from one that was genuinely starved, and the penalty must also be large enough to outweigh the extra allocation won by lying. We do not assume this argument holds. We include a scoring-rule mechanism, scored on censored sales and with the penalty weight tuned, as a baseline, and we also test ranking plus scoring combined. If scoring alone matches ranking at moderate censoring, the honest conclusion is that ranking helps only under heavy stockouts. We will report where that crossover lies.

Why not just use one central auditor, such as a critic LLM as in Guo et al., instead of a regional editorial tier? A central estimator that applies the same isotonic step without regional aggregation is a direct ablation. If it matches the two-tier version at lower delay, the editorial hierarchy adds nothing beyond the estimator, and we will say so. An LLM critic that reads reports and flags suspicious ones is also run, because it is the obvious modern alternative. It has no incentive guarantee, and The Limits of Optimal Pricing in the Dark suggests that flexible detectors are exactly what deceivers learn to exploit.

Proposed Experiments

The main testbed uses the SupplyGraph data, which covers 41 products with daily sales orders and deliveries from January to August 2023. We construct true demand paths from the sales orders and the node's signals from the delivery series plus controlled noise. We impose capacity shortfalls of 10–40% and assign product subsets to synthetic retailers. Metrics are total chain profit, fill rate, forecast error of $\hat\theta$, retailer information rent, manipulation gain under best-response ranking, and the bullwhip ratio (variance of orders over variance of demand) when the allocation repeats over periods.

The serial four-stage beer game from A Deep Q-Network for the Beer Game with Partial Information serves as a robustness check against irrational partners using Sterman's behavioral ordering rule. In that work the learned agent beat an optimized base-stock policy by 23%, which shows partner behavior matters. Here we check whether ranking still helps when retailers are erratic rather than strategic. The LLM experiments use the InvAgent and Moros simulators with Merton jump-diffusion demand.

The decisive check has three parts that must all hold for the proposal's main claim to stand. First, under the convexified allocation rule the measured manipulation gain should be near zero. Under proportional allocation with newsvendor retailers it should be clearly positive, which would confirm that the theory predicts where the mechanism fails. Second, ranking should beat the signals-only ablation by a margin that grows as the node's signals get noisier, since that is exactly when private information is worth more. Third, ranking should match the per-product screening baseline within its reported optimality gap while using no side payments.