Variation Control and Evaluation for Generative Slate Recommendations

Shuchang Liu, Fei Sun, Yingqiang Ge, Changhua Pei, Yongfeng Zhang

Abstract

Slate recommendation generates a list of items as a whole instead of ranking each item individually, so as to better model the intra-list positional biases and item relations. In order to deal with the enormous combinatorial space of slates, recent work considers a generative solution so that a slate distribution can be directly modeled. However, we observe that such approaches—despite their proved effectiveness in computer vision—suffer from a trade-off dilemma in recommender systems: when focusing on reconstruction, they easily over-fit the data and hardly generate satisfactory recommendations; on the other hand, when focusing on satisfying the user interests, they get trapped in a few items and fail to cover the item variation in slates. In this paper, we propose to enhance the accuracy-based evaluation with slate variation metrics to estimate the stochastic behavior of generative models. We illustrate that instead of reaching to one of the two undesirable extreme cases in the dilemma, a valid generative solution resides in a narrow “elbow” region in between. And we show that item perturbation can enforce slate variation and mitigate the over-concentration of generated slates, which expand the “elbow” performance to an easy-to-find region. We further propose to separate a pivot selection phase from the generation process so that the model can apply perturbation before generation. Empirical results show that this simple modification can provide even better variance with the same level of accuracy compared to post-generation perturbation methods.

Introduction

In modern recommender systems, items are rarely shown in isolation; they are presented as a slate (e.g., a 5-item list or a 2x2 grid). Traditional "Learning-to-Rank" (LTR) methods rank items individually and pick the top $K$. However, this greedy approach ignores how items in a list interact—for example, two great items might be redundant if shown together. To capture these interactions, researchers have turned to generative models like Conditional Variational Auto-Encoders (CVAE) to model the distribution of entire slates.

The Reconstruction-Concentration Dilemma (RCD)

Generative models for slate recommendation face a major hurdle: the Reconstruction-Concentration Dilemma (RCD). This is a trade-off between matching the training data distribution and satisfying user interests.

  • Over-reconstruction: If the model focuses too much on matching training data (low reconstruction loss), it generates slates with high variance but poor recommendation quality.
  • Over-concentration: If the model focuses too much on user interests (high KLD), it gets stuck in a "mode collapse," repeatedly suggesting the same few high-performing slates and failing to explore the vast combinatorial space of possible item combinations.
Figure 5: Visualization of the Reconstruction-Concentration Dilemma during training, showing clusters of over-concentration and over-reconstruction.

The authors argue that the "sweet spot" for these models—the Elbow Case—is frustratingly narrow and hard to find using standard accuracy metrics alone.

Methodology: Pivot-CVAE

To address the RCD, the authors propose the Pivot-CVAE framework. Instead of generating a slate all at once, they split the process into two phases:

  1. Pivot Selection: A model selects a single "pivot" item for a fixed position.
  2. Slate Completion: A second model generates the rest of the slate conditioned on that pivot and the latent user state.
Figure 1: Overall architecture of the generative framework comparing the standard List-CVAE decoder with the proposed Pivot-CVAE decoder.

By applying perturbation (randomly swapping the pivot) before the rest of the slate is generated, the model can explore new item combinations. Crucially, because the completion model is trained to fill in the slate based on the pivot, it can "correct" for a noisy pivot choice, maintaining higher accuracy than methods that perturb the list after it's fully generated.

Evaluation and Results

The researchers evaluated their approach on real-world datasets (YOOCHOOSE, MovieLens 100K) and simulated environments. They introduced new variation metrics, decomposing total item variance into slate-mean variance (how much slates differ) and intra-slate variance (diversity within a slate).

Key findings include:

  • Pivot-CVAE consistently outperformed standard List-CVAE and LTR baselines in balancing accuracy and variety.
  • Item Coverage: In simulated environments with high item interaction, Pivot-CVAE achieved an Item Coverage of 0.227, compared to just 0.005 for the standard List-CVAE.
  • Robustness: The two-phase approach allowed the model to maintain high accuracy even when the pivot was heavily perturbed, proving that the completion phase successfully models complex intra-list relationships.

Conclusion

The Pivot-CVAE framework effectively mitigates the Reconstruction-Concentration Dilemma in generative slate recommendation. By separating pivot selection from slate completion and introducing variation-aware metrics, the model achieves a better balance of accuracy and diversity. Future work will explore extending this approach to GANs and non-linear user response simulations.

Summary updated

Abstract

Slate recommendation generates a list of items as a whole instead of ranking each item individually, so as to better model the intra-list positional biases and item relations. In order to deal with the enormous combinatorial space of slates, recent work considers a generative solution so that a slate distribution can be directly modeled. However, we observe that such approaches—despite their proved effectiveness in computer vision—suffer from a trade-off dilemma in recommender systems: when focusing on reconstruction, they easily over-fit the data and hardly generate satisfactory recommendations; on the other hand, when focusing on satisfying the user interests, they get trapped in a few items and fail to cover the item variation in slates. In this paper, we propose to enhance the accuracy-based evaluation with slate variation metrics to estimate the stochastic behavior of generative models. We illustrate that instead of reaching to one of the two undesirable extreme cases in the dilemma, a valid generative solution resides in a narrow “elbow” region in between. And we show that item perturbation can enforce slate variation and mitigate the over-concentration of generated slates, which expand the “elbow” performance to an easy-to-find region. We further propose to separate a pivot selection phase from the generation process so that the model can apply perturbation before generation. Empirical results show that this simple modification can provide even better variance with the same level of accuracy compared to post-generation perturbation methods.
BibTeX
@inproceedings{liu2021variation,
  author = {Shuchang Liu and Fei Sun and Yingqiang Ge and Changhua Pei and Yongfeng Zhang},
  title = {{Variation Control and Evaluation for Generative Slate Recommendations}},
  booktitle = {The Web Conference},
  year = {2021},
  url = {https://openreview.net/forum?id=LKLE91VDLxz},
}

Loading related content...

Figure