A Structured Latent Variable Recurrent Network With Stochastic Attention For Generating Weibo Comments

Shijie Yang, Liang Li, Shuhui Wang, Weigang Zhang, Qingming Huang, Qi Tian

Introduction

Generating engaging and diverse comments for social media platforms like Sina Weibo remains a significant challenge for natural language processing. Traditional sequence-to-sequence (Seq2Seq) models, while effective at learning patterns, frequently fall into the "safe response" trap. Because these models are optimized for maximum likelihood, they tend to output high-frequency, generic phrases such as "I don't know" or "Me too," which lack the nuance and variety of human discourse.

The SARN Framework

To address this, we propose the Structured Latent Variable Recurrent Network (SARN). The core philosophy behind SARN is that human language is inherently hierarchical. When responding to a post, a human first establishes a high-level intent or topic (discourse) and then selects specific words to express that intent.

Comparison between standard Seq2Seq models and the proposed SARN model, highlighting discourse-level and word-level diversity.

SARN implements this hierarchy through two levels of latent variables:

  1. Discourse-level ($z_d$): A global variable capturing the overall style, topic, and intent of the response.
  2. Word-level ($z_w$): A sequence of variables that capture local variations in word choice at each generation step. These variables maintain temporal dependencies, ensuring that the "trajectory" of word selection remains coherent.

Stochastic Attention

A second major innovation in SARN is the introduction of stochastic attention. Standard attention mechanisms are deterministic; given a specific input, they always focus on the same parts of the source text. In contrast, SARN’s attention is driven by the word-level latent variables. This allows the model to "interpret" the same input post in multiple valid ways by stochastically focusing on different keywords at each step, facilitating a robust 1-to-n mapping between posts and potential responses.

Experimental Results and Analysis

The model was evaluated on a dataset of 1 million post-comment pairs from Sina Weibo. To ensure a diverse training environment, the researchers used Tf-Idf cosine distance to group similar comments for each post.

In human evaluations, SARN achieved a significantly higher Acceptability ratio (69.3%) compared to baselines. More importantly, it nearly doubled the diversity score of standard Seq2Seq models (2.95 vs. 1.51).

Qualitative examples of generated comments showing the impact of hyperparameters on diversity and acceptability.

As shown in the qualitative analysis, the model's performance is sensitive to the weights of the latent variables (controlled by hyperparameters $\alpha$ and $\beta$). While increasing these weights boosts diversity, it can eventually impact grammatical correctness, highlighting a critical "Diversity-Grammar Trade-off" that must be balanced for optimal performance.

Conclusion

By integrating hierarchical latent variables with a stochastic attention mechanism, SARN successfully mimics the multi-level decision-making process of human language generation. This approach significantly mitigates the "safe response" problem, producing social media comments that are both contextually relevant and linguistically diverse. Future work will focus on refining the balance between these hierarchical levels to further enhance long-term coherence.

Summary updated

Abstract

Building intelligent agents to generate realistic Weibo comments is challenging. For such realistic Weibo comments, the key criterion is improving diversity while maintaining coherency. Considering that the variability of linguistic comments arises from multi-level sources, including both discourse-level properties and word-level selections, we improve the comment diversity by leveraging such inherent hierarchy. In this paper, we propose a structured latent variable recurrent network, which exploits the hierarchical-structured latent variables with stochastic attention to model the variations of comments. First, we endow both discourse-level and word-level latent variables with hierarchical and temporal dependencies for constructing multi-level hierarchy. Second, we introduce a stochastic attention to infer the key-words of interest in the input post. As a result, diverse comments can be generated with both discourse-level properties and local-word selections. Experiments on open-domain Weibo data show that our model generates more diverse and realistic comments.
BibTeX
@inproceedings{yang2020structured,
  author = {Shijie Yang and Liang Li and Shuhui Wang and Weigang Zhang and Qingming Huang and Qi Tian},
  title = {{A Structured Latent Variable Recurrent Network With Stochastic Attention For Generating Weibo Comments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/548},
}
Figure