Multi-Scale Distribution Deep Variational Autoencoder for Explanation Generation

Zefeng Cai, Linlin Wang, Gerard de Melo, Fei Sun, Liang HeACL 2022 · OpenReview

Abstract

Generating explanations for recommender systems is essential for improving their transparency, as users often wish to understand the reason for receiving a specified recommendation. Previous methods mainly focus on improving the generation quality, but often produce generic explanations that fail to incorporate user and item specific details. To resolve this problem, we present Multi-Scale Distribution Deep Variational Autoencoders (MVAE). These are deep hierarchical VAEs with a prior network that eliminates noise while retaining meaningful signals in the input, coupled with a recognition network serving as the source of information to guide the learning of the prior network. Further, the Multi-scale distribution Learning Framework (MLF) along with a Target Tracking Kullback-Leibler divergence (TKL) mechanism are proposed to employ multi KL divergences at different scales for more effective learning. Extensive empirical experiments demonstrate that our methods can generate explanations with concrete input-specific contents.

Introduction

When a streaming service or restaurant app provides a recommendation, transparency is key to user trust. While modern systems attempt to generate "free-text" explanations (e.g., "The atmosphere is relaxing"), they frequently suffer from genericness.

The root of this problem lies in how models represent users and items. Most systems rely on "ID embeddings"—mathematical vectors assigned to each user and product. However, because most users leave very few reviews, these embeddings are often sparse and "noisy." When a model encounters a noisy, poorly trained embedding, it defaults to generating safe, one-size-fits-all sentences. To generate high-quality, specific explanations, a model must distinguish meaningful signals from this inherent noise.

The MVAE Architecture

The proposed Multi-Scale Distribution Deep Variational Autoencoder (MVAE) addresses the noise problem through a hierarchical structure. The architecture is divided into distinct phases for training and inference to bridge the gap between known ground-truth data and predictive generation.

The MVAE architecture distinguishes between (a) training, where a Recognition Network acts as a teacher, and (b) testing, where a Prior Network generates explanations from IDs alone.

As shown in the architecture diagram, the model consists of three primary components:

  1. Recognition Network: Used during training, this network extracts "fine-grained" information from actual ground-truth review text. It serves as a teacher for the rest of the system.
  2. Prior Network: This network only has access to the User ID, Item ID, and rating. It learns to approximate the fine-grained signals of the Recognition Network, essentially learning to "predict" the details of a review from IDs alone.
  3. Reconstruction Network: A GPT-based decoder that transforms latent signals into readable, natural language sentences.

Technical Innovations: MLF and TKL

To optimize this hierarchical VAE, the researchers introduced two key mechanisms:

  • Multi-scale distribution Learning Framework (MLF): Rather than using a single "bottleneck" layer, MLF utilizes multiple layers of distributions at different scales. This allows the model to capture both broad semantic themes and minute, specific details.
  • Target Tracking KL (TKL): VAEs often suffer from "posterior collapse," where the model ignores latent variables. TKL forces the Prior network to "track" the Recognition network’s distribution while remaining close to a standard normal distribution. This ensures the Prior network remains effective during testing when ground-truth text is unavailable.

Experimental Insights: The Impact of Noise

A core finding of the research is that noise is the primary enemy of specificity. By analyzing the cosine similarity of user embeddings, the authors demonstrated that users with fewer training samples have embeddings that are essentially random noise.

Analysis showing that as the number of training samples increases, the stability (cosine similarity) of user embeddings improves, reducing the noise that leads to generic explanations.

By forcing input through the VAE's hierarchical bottleneck, MVAE acts as a filter, discarding this noise and retaining only the essential signals required to reconstruct a specific explanation.

Results and Conclusion

The MVAE model was evaluated on three major datasets: Yelp (restaurants), Amazon (Movies & TV), and TripAdvisor (hotels). The datasets were specifically filtered to include "challenging samples"—users and items with very few reviews—to test the model's robustness against sparse data.

Key performance highlights include:

  • Significant Metric Gains: MVAE improved BLEU-4 scores by 95.91% on Yelp and 113.32% on TripAdvisor compared to previous state-of-the-art models like PETER and NETE.
  • Qualitative Specificity: While baseline models produced generic phrases like "The service is great," MVAE generated detailed descriptions such as "The staffs are knowledgeable and the customer service is impressive."
  • Robustness: The model showed its most significant relative improvements on the most sparse data, proving its effectiveness at overcoming the noisy embedding problem.

While MVAE represents a major step forward in generating specific explanations, future work will look toward "cold-start" scenarios where users have no history at all, potentially incorporating external knowledge graphs to supplement sparse ID data.

Summary updated

Abstract

Generating explanations for recommender systems is essential for improving their transparency, as users often wish to understand the reason for receiving a specified recommendation. Previous methods mainly focus on improving the generation quality, but often produce generic explanations that fail to incorporate user and item specific details. To resolve this problem, we present Multi-Scale Distribution Deep Variational Autoencoders (MVAE).These are deep hierarchical VAEs with a prior network that eliminates noise while retaining meaningful signals in the input, coupled with a recognition network serving as the source of information to guide the learning of the prior network. Further, the Multi-scale distribution Learning Framework (MLF) along with a Target Tracking Kullback-Leibler divergence (TKL) mechanism are proposed to employ multi KL divergences at different scales for more effective learning. Extensive empirical experiments demonstrate that our methods can generate explanations with concrete input-specific contents.
BibTeX
@inproceedings{cai2022multi,
  author = {Zefeng Cai and Linlin Wang and Gerard de Melo and Fei Sun and Liang He},
  title = {{Multi-Scale Distribution Deep Variational Autoencoder for Explanation Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=4yIRstTAZV},
}
Figure