Abstract
This paper presents a large-scale corpus for non-task-oriented dialogue response selection, which contains over 27K distinct prompts and more than 82K responses collected from social media. To annotate this corpus, we define a 5-grade rating scheme: bad, mediocre, acceptable, good, and excellent, according to relevance, coherence, informativeness, interestingness, and the potential to move a conversation forward. Experimental results confirm that the proposed corpus is helpful in training response selection models, significantly outperforming unsupervised baselines.
Introduction
Building chatbots capable of natural "chitchat" remains a significant challenge in AI. Unlike task-oriented agents with specific goals, non-task-oriented systems must handle diverse topics while maintaining engagement. While massive social media datasets (e.g., Weibo) provide abundant training material, they are often noisy, containing off-topic content or "versatile" responses—generic replies like "Me too!" that are technically correct but conversationally dull. This paper addresses the need for high-quality, manually labeled data to help models distinguish between safe, mediocre replies and truly excellent, informative ones.
Methods: The 5-Grade Rating Scheme
The researchers developed a rigorous human annotation pipeline to label 82,010 prompt-response pairs. Moving beyond binary "good/bad" labels, they implemented a nuanced 5-grade rating scheme to capture the quality of dialogue more effectively:
- Bad (R1): Nonsense or completely off-topic.
- Mediocre (R2): Mentions keywords but is incoherent or merely echoes the prompt.
- Acceptable (R3): Meaningful and relevant, but generic ("versatile") or context-limited.
- Good (R4): Coherent and relevant, but simple and lacks proactive engagement.
- Excellent (R5): Informative, interesting, and proactively leads the conversation forward.
Dataset Characteristics
The corpus was curated from popular Chinese social platforms, including Tieba, Zhidao, Douban, and Weibo. After filtering for sensitive content and removing knowledge-dependent prompts (e.g., factual queries), the final dataset consists of 27,383 distinct prompts. The annotation process achieved a high Cohen’s kappa of 80.0%, indicating strong agreement between human annotators.
Analysis of the data revealed that 48.9% of responses were categorized as "Acceptable" (Rating 3). This highlights the prevalence of generic replies in social media data and underscores the importance of fine-grained labeling to allow models to penalize mediocrity and favor engagement.
Experimental Results
To validate the corpus, the researchers compared unsupervised models (Cosine Similarity, BM25) against supervised models (SVMRank, GBDT, BiLSTM, CNN). Key findings include:
- Supervised Superiority: Supervised models consistently outperformed unsupervised baselines across all metrics, including P@1, MAP, and MRR.
- Neural Performance: When evaluated against the strictest standard ("Excellent" only), the CNN model achieved a Precision@1 of 70.5%, significantly higher than the 53.8% achieved by the best unsupervised model (BM25).
- The Challenge of Excellence: All models experienced a performance drop as the evaluation criteria became stricter (e.g., moving from a "Cut@3" threshold to "Cut@5"). This suggests that identifying truly engaging responses is a significantly harder task than identifying merely relevant ones.
Conclusion
This work provides the first large-scale, manually annotated Chinese corpus for non-task-oriented dialogue. By providing a 5-grade rating scale, the dataset enables the development of models that prioritize informative and engaging conversation over generic "safe" replies. While currently focused on single-turn response selection, future work will explore extending this approach to multi-turn dialogues and applying the corpus to improve generative dialogue models.
Summary updated