Introduction
Generating engaging and diverse comments for social media platforms like Sina Weibo remains a significant challenge for natural language processing. Traditional sequence-to-sequence (Seq2Seq) models, while effective at learning patterns, frequently fall into the "safe response" trap. Because these models are optimized for maximum likelihood, they tend to output high-frequency, generic phrases such as "I don't know" or "Me too," which lack the nuance and variety of human discourse.
The SARN Framework
To address this, we propose the Structured Latent Variable Recurrent Network (SARN). The core philosophy behind SARN is that human language is inherently hierarchical. When responding to a post, a human first establishes a high-level intent or topic (discourse) and then selects specific words to express that intent.
SARN implements this hierarchy through two levels of latent variables:
- Discourse-level ($z_d$): A global variable capturing the overall style, topic, and intent of the response.
- Word-level ($z_w$): A sequence of variables that capture local variations in word choice at each generation step. These variables maintain temporal dependencies, ensuring that the "trajectory" of word selection remains coherent.
Stochastic Attention
A second major innovation in SARN is the introduction of stochastic attention. Standard attention mechanisms are deterministic; given a specific input, they always focus on the same parts of the source text. In contrast, SARN’s attention is driven by the word-level latent variables. This allows the model to "interpret" the same input post in multiple valid ways by stochastically focusing on different keywords at each step, facilitating a robust 1-to-n mapping between posts and potential responses.
Experimental Results and Analysis
The model was evaluated on a dataset of 1 million post-comment pairs from Sina Weibo. To ensure a diverse training environment, the researchers used Tf-Idf cosine distance to group similar comments for each post.
In human evaluations, SARN achieved a significantly higher Acceptability ratio (69.3%) compared to baselines. More importantly, it nearly doubled the diversity score of standard Seq2Seq models (2.95 vs. 1.51).
As shown in the qualitative analysis, the model's performance is sensitive to the weights of the latent variables (controlled by hyperparameters $\alpha$ and $\beta$). While increasing these weights boosts diversity, it can eventually impact grammatical correctness, highlighting a critical "Diversity-Grammar Trade-off" that must be balanced for optimal performance.
Conclusion
By integrating hierarchical latent variables with a stochastic attention mechanism, SARN successfully mimics the multi-level decision-making process of human language generation. This approach significantly mitigates the "safe response" problem, producing social media comments that are both contextually relevant and linguistically diverse. Future work will focus on refining the balance between these hierarchical levels to further enhance long-term coherence.
Summary updated