Where 2D Layout Simulators Lie: Pessimistic Training for Grid and Carousel Policies

Recommendation pages are rarely a single list. Shops show grids, streaming services stack horizontal carousels, and mobile apps mix vertical feeds with swipeable rows. How likely a user is to look at an item therefore depends on its row, its column, and its neighbours, not on one rank number. This is two-dimensional position bias. The best current methods for arranging such pages are reinforcement-learning agents that learn by trial and error. Because live trial and error is too costly, they are trained against a simulated user who scans the page by a hand-written rule, such as a Z-shaped path, a preference for middle columns, or a habit of skipping whole rows. That puts recommendation in the same position as robotics and autonomous driving: a policy is optimized in a simulator and then deployed on reality. In those fields the resulting sim-to-real gap is well documented, and optimizers are known to exploit simulator quirks rather than learn the real dynamics.

This note makes one claim and turns it into a method. The damaging part of the 2D simulator gap is not random miscalibration. It is concentrated in placements the logged system never made, where many different scanning models fit the historical clicks equally well. A layout agent is rewarded for moving items into exactly those unconstrained slots. We propose to fit a set of structured 2D browsing models that all agree with real logs. We then train the layout policy against the worst plausible member of that set. To test the idea, we build a real-data protocol that fits simulators on one region of layout space and evaluates on another, so the sim-to-real gap becomes measurable without live traffic.

Core Claim

Historical logs identify how users examine a page only along the placements the logging policy actually produced. Suppose the old system always put popular items top-left. Then a "top-left golden triangle" model and a "middle-column Gaussian" model may predict the same logged clicks, yet they disagree sharply about a page that puts the best item in row two, column three. A layout agent trained in either model will learn to exploit whatever that model says about unseen placements.

Two predictions follow. First, the transfer loss of a simulator-trained layout policy should grow with how far its chosen placements lie from the logging support. It should not simply track the simulator's likelihood on held-out logs. Second, a policy that is conservative exactly where log-consistent browsing models disagree should transfer better than two alternatives: a policy trained against the single best-fitting simulator, and a policy trained against uniformly randomized simulators.

The mechanism borrows from sim-to-real work in two ways. The first borrowing is a diagnosis. Networks trained on simulated particle jets learn artifacts of the simulation code instead of physics (Jet substructure classification in high-energy physics with deep neural networks; Boosted Jet Tagging with Jet-Images and Deep Neural Networks). We expect the same of layout agents facing hand-coded scanning rules.

The second borrowing is a design constraint. Adversarially Masking Synthetic to Mimic Real shows that when synthetic data are clean and real data are noisy, a domain discriminator latches onto the noise itself. Alignment then fails unless structured noise is injected into the synthetic side. The 2D analogue is that simulated sessions follow deterministic scan paths, while real sessions contain accidental clicks, back-tracking, and appearance-driven jumps. The browsing model family must therefore contain those effects explicitly. Otherwise log-consistency checks will reject every simulator for trivial reasons.

What Exists and What It Leaves Open

Two papers already train layout agents in synthetic 2D user environments. Reinforcement Re-ranking with 2D Grid-based Recommendation Panels casts filling a 2×3 grid as a Markov decision process called Panel-MDP. In it, the agent decides for each candidate whether to show it and in which slot. It is trained with proximal policy optimization against a simulator with hand-set middle bias and row skipping, and it is evaluated separately on offline e-commerce logs. Tile Networks trains a pointer-attention encoder-decoder with REINFORCE across Row-wise, Column-wise, Z-shape, and a log-derived "Real-Env" simulator, and shows that the model learns a viewing-priority map matching whichever rule the simulator encodes. Neither paper measures what happens when a policy trained under one browsing rule meets users who follow another. The Tile Networks finding is exactly the symptom we expect: the policy fits the simulator's geometry. We reuse both agents unchanged and change only what they are trained against.

The descriptive evidence on 2D browsing supplies the model family. Web image grids show Z-shaped scanning (Constructing an Interaction Behavior Model for Web Image Search), a Gaussian middle-column bias, row skipping, and slower vertical decay (1.06 for image search versus 1.48 for text web search) (Grid-based Evaluation Metrics for Web Image Search). Multi-block mobile pages produce F-shaped paths (An F-shape Click Model for Information Retrieval on Multi-block Mobile Pages). Map interfaces decay radially from the viewport centre (Learning to Rank for Maps at Airbnb). Carousel interfaces follow a top-left diagonal (Offline Evaluation of Recommender Systems in a User Interface With Multiple Carousels).

These papers mostly treat their pattern as the model to be calibrated. We treat them instead as components of a mixture whose weights the logs only partly pin down. The grid-metrics paper also flags visual salience ("appearance bias") as an unmodeled confounder. We include it as a per-item term, following the lesson from the masking paper that unmodeled real-world noise must be put into the simulator.

The carousel evaluation papers (Evaluating Recommendations in a User Interface With Multiple Carousels; Measuring the Ranking Quality of Recommendations in a Two-Dimensional Carousel Setting) motivate the whole enterprise. They show that the choice of 2D discount reorders which algorithms look best. EASE^R drops 4 positions on MovieLens 20M and 6 on Netflix, and FunkSVD rises 7 on Netflix. The authors say their deterministic discounts are unvalidated against real stochastic users. If the discount model alone can flip conclusions, then a policy optimized against one discount model inherits that fragility.

Research Program

The browsing model family is a stochastic examination model over slots $(m, n)$. It mixes interpretable geometric components: row-major Z-scanning, column-major scanning, F-shaped block scanning, a column Gaussian, top-left diagonal decay, and radial decay. It adds a row-skip continuation probability, a per-item salience term computed from thumbnail features, and an explicit noise channel for accidental clicks and back-jumps. Every member is a proper click simulator, so either agent can train inside it.

The first stage is building the plausible set. We fit the family to logs by maximum likelihood, then keep every parameter setting whose held-out log-likelihood falls within a tolerance $\epsilon$ of the best. We sample this set with an ensemble of independently initialized fits plus perturbation along directions where the likelihood is flat. We also train a small discriminator that separates simulated sessions from real ones under the logging policy. It is used only as a check that the noise channel has removed trivial "too clean" differences, following the masking paper's warning. It is not used as the main fitting signal.

The second stage is pessimistic training. We write $\pi$ for the layout policy, $\mathcal{U}_\epsilon$ for the plausible set of browsing models, and $R_u(\pi)$ for the expected clicks of $\pi$ in model $u$. We train

$$ \max_{\pi} \; \mathbb{E}_{u \sim \mathcal{U}_\epsilon}\big[R_u(\pi)\big] \;-\; \lambda\, \mathrm{Std}_{u \sim \mathcal{U}_\epsilon}\big[R_u(\pi)\big], $$

where $\lambda \ge 0$ trades expected reward against disagreement across plausible users. Setting $\lambda = 0$ recovers ensemble randomization. A single $u$ recovers standard calibrated-simulator training. The penalty is large exactly on placements the logs cannot adjudicate, so the policy is pushed to reach high reward with placements whose value the data actually supports.

The third stage is the diagnostic, which is a contribution in its own right. For any simulator-trained policy we report its simulator reward, its reward under held-out real behaviour, and its placement distance from the logging support. The distance is the divergence between the policy's slot distribution for high-relevance items and that of the logs. The prediction to check is that transfer loss grows with this distance.

Baselines

The naive reference is the logged production arrangement where available, together with "fill slots row-major in relevance order." This is what a system that ignores 2D bias does. Any learned policy that cannot beat it under held-out real behaviour has learned only simulator artifacts.

The strongest supervised competitor is miDNN as benchmarked in the Panel-MDP paper, which reports AUC 0.7524 on the e-commerce 2×3 panel data. We rerun it on our splits rather than quoting that number across protocols.

The strongest existing layout agents are Panel-MDP, with a reported "Select & Re-org" AUC of 0.7980 and average reward 0.0179 on the real e-commerce dataset (Reinforcement Re-ranking with 2D Grid-based Recommendation Panels), and Tile Networks, with a reported NDCG of 0.794 in Real-Env on Alibaba data versus 0.765 for Deconv (Tile Networks). Each is trained in its original hand-coded simulator. They are the most direct competitors because our method changes only their training environment. Each agent also gets a stronger variant trained in the single maximum-likelihood simulator from our family. That variant separates "better model family" from "pessimism over the family."

For fairness, all variants use identical candidate pools, identical agent architectures and PPO/REINFORCE hyperparameter search budgets, identical numbers of simulated environment steps, and identical held-out evaluation. The ablations are uniform domain randomization over the same family with no log-consistency filter, the plausible ensemble with $\lambda = 0$, the model family without the salience term, and the family without the noise channel. The comparison with $\lambda = 0$ isolates pessimism. The comparison with uniform randomization isolates log-consistency. Beating the baselines means a statistically significant improvement in held-out real reward under the support-shift split (defined below) over the single-calibrated-simulator variant of the same agent, with no loss on the in-support split.

Why not just…?

Why not just fit the best click model to the logs and train in it? This is the real threat, and it is the single-calibrated-simulator baseline above. Our argument is identification. Maximum likelihood picks one point from a set of models that are equivalent on the logging support, and the agent then exploits that point's arbitrary off-support predictions. If the plausible set turns out to be narrow, meaning the logs pin down geometry well because production layouts varied enough, pessimism will add nothing and the calibrated simulator will match us. We will report plausible-set width per dataset and treat a narrow set as a negative result for the method. The diagnostic would still stand.

Why not just use 2D inverse propensity scoring on real logs? Propensity weighting reweights logged clicks by estimated examination probability. It cannot value placements the logger never made, because those placements have zero or near-zero propensity. That is exactly where layout agents want to go. The carousel ranking flips (Evaluating Recommendations in a User Interface With Multiple Carousels) show how strongly logs are shaped by existing shelves. We include a propensity-weighted supervised learner as an extra baseline. We expect it to do well in-support and fail on the support-shift split.

Why not just tune N2DCG-style discounts with a small eye-tracking study? A tuned deterministic discount is a single point in our family and inherits the same identification problem. The grid-metrics work also shows that browsing is context-dependent: decay rates differ between image and text search, and row skipping appears. Preference-based Evaluation Metrics for Web Image Search reports that fixed positional metrics trail a pairwise preference metric by 23% in Pearson correlation with expert SERP preferences. We use the eye-tracking data as ground truth for evaluation rather than as a tuning set, which tests this alternative directly.

Why not just use uniform domain randomization, as in robotics? Randomization over geometries that contradict the logs wastes capacity on impossible users and yields timid policies. Log-consistency filtering is the ablation that decides this.

Proposed Experiments

The central protocol is a real-data support-shift split. On ContentWise Impressions, which records which carousels and positions were actually displayed, and on the e-commerce 2×3 panel data, we partition sessions by placement regime. For example, we separate sessions where high-relevance items appeared in the top-left region from sessions where they appeared elsewhere. We fit simulators only on the first regime and evaluate policies by replaying against logged behaviour in the second regime. Evaluation is restricted to decisions the second regime covers, using the same held-out counterfactual estimator for every method. This produces a genuine, measurable sim-to-real gap from real users without live traffic.

The Alibaba whole-page data gives a second instance, following the Tile Networks "Real-Env" construction. On the Sogou image-search resources described in Grid-based Evaluation Metrics for Web Image Search, we fit the family to the 100,000-session click log. We then score each fitted model's examination predictions against the 40-participant eye-tracking fixations, which act as ground truth. The reference point is that paper's row-skipping model, which improved log-likelihood by 13.5% over linear rank-biased precision. We also report a synthetic transfer matrix: train under each hand-coded rule, test under every other. This reproduces the Tile Networks setting with the missing off-diagonal cells filled in.

Metrics are held-out click reward and AUC on the shift split, the simulator-minus-real transfer gap, and N2DCG on carousel data. For image grids we add log-likelihood of eye-tracking fixations. The diagnostic plots transfer gap against placement distance from the logging support.

Three decisive checks follow. The first is whether transfer gap tracks placement distance more strongly than it tracks simulator held-out likelihood. If not, the core claim is wrong. The second is whether pessimism over the plausible set beats the single calibrated simulator on the shift split while tying it in-support. The third is whether removing the salience or noise terms makes the sim-versus-real discriminator trivially accurate and degrades transfer, which would confirm that the masking-paper lesson carries over to users.