Overview of the NLPCC 2017 Shared Task: Chinese News Headline Categorization

Xipeng Qiu, Jingjing Gong, Xuanjing Huang

Introduction

With the rapid expansion of internet and social media content, the manual categorization of news articles and web text has become an insurmountable task. Machine learning offers a scalable solution, yet its effectiveness relies heavily on high-quality, labeled datasets for training and evaluation. This paper addresses the specific challenge of short text classification through the lens of the NLPCC 2017 Shared Task.

News headlines are uniquely challenging due to their brevity; they lack the rich context found in full-length articles, making it difficult for algorithms to accurately determine topics. To benchmark progress in this area, the authors organized a shared task to assess how modern algorithms handle brief Chinese text snippets.

Methodology and Evaluation

The task is framed as a multi-class classification problem. Headlines are segmented into words using the jieba tokenizer and mapped to one of 18 predefined categories. To establish a performance baseline, the authors implemented three standard deep learning architectures:

  • Neural Bag-of-Words (NBoW): A simple model focusing on word occurrences regardless of order.
  • Convolutional Neural Networks (CNN): Designed to extract local spatial features and phrases.
  • Long Short-Term Memory (LSTM): A recurrent network capable of capturing sequential dependencies.

Participating teams were evaluated based on Macro-averaged Precision, Recall, F1-score, and Accuracy.

Dataset Characteristics

The authors compiled a new dataset of Chinese news headlines from major portals such as Toutiao and Sina. The dataset comprises 228,000 examples, partitioned into training, development, and test sets.

Summary of dataset size and average text lengths.

The data is distributed across 18 categories, including entertainment, sports, tech, finance, and military. Most categories contain 10,000 training samples, while a few smaller categories have 4,000. A defining characteristic of this dataset is the extreme brevity of the entries, with an average length of approximately 22 characters or 13 words per headline.

Results and Insights

The shared task attracted 32 participating teams, reflecting significant interest in non-English short-text classification.

Baseline Performance

Surprisingly, the simplest baseline model, NBoW, outperformed the more complex CNN and LSTM architectures. This suggests that for very short, topic-dense texts like headlines, keyword presence is often as informative as complex syntactic structures.

Performance metrics for the baseline models.

Competition Highlights

The top-performing participant (P1) achieved a Macro F1 of 0.830 and an Accuracy of 0.829, significantly exceeding the baselines. The top 10 teams all surpassed a 0.800 Macro F1, demonstrating that specialized architectures and ensemble methods can effectively extract meaning even from minimal context.

Conclusion

The NLPCC 2017 Shared Task highlights the potential for automated Chinese news categorization. While simple baselines provide a strong starting point, the competition results show that there is still room for optimization using more advanced techniques. By open-sourcing this dataset, the authors have provided a valuable resource for the continued development of neural architectures tailored for ultra-short Chinese text.

Summary updated

BibTeX
@inproceedings{qiu2017overview,
  author = {Xipeng Qiu and Jingjing Gong and Xuanjing Huang},
  title = {{Overview of the NLPCC 2017 Shared Task: Chinese News Headline Categorization}},
  booktitle = {NLPCC},
  year = {2017},
  url = {https://openreview.net/forum?id=J1BoURjY4Me},
}
Figure