MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization

Canwen Xu, Jiaxin Pei, Hongtao Wu, Yiyu Liu, Chenliang Li

Introduction

The rapid advancement of Natural Language Processing (NLP) has been fueled by large-scale datasets. However, most existing datasets are "siloed," designed for a single task like question answering (QA), summarization, or classification. This specialization creates a bottleneck for Multi-Task Learning (MTL). When a model is trained on multiple disparate datasets, it is difficult to determine if performance gains stem from better general representations or simply from exposure to more diverse data distributions.

To bridge this gap, researchers have introduced MATINF, the first large-scale Chinese dataset jointly labeled for three major NLP tasks. By providing a single source of data for multiple labels, MATINF offers a "fair stage" for evaluating MTL models without the confounding variables of differing data distributions.

The MATINF Dataset

MATINF is a massive corpus of 1.07 million unique question-answer pairs crawled from a popular Chinese maternity and childcare website. Each entry consists of a question (Q), a detailed situational description (D), human-labeled categories (C), and the "best answer" (A) chosen by the user.

Comparison of traditional multi-task learning paradigms (a) versus the MATINF approach (b), alongside layer (c) and module (d) sharing strategies.

Unlike general news datasets, MATINF-C focuses on a single domain, making classification challenging due to subtle semantic differences between categories like "infant nutrition" and "infant feeding." Furthermore, the summarization task (MATINF-SUMM) utilizes noisy, user-generated content, reflecting real-world social media dynamics rather than formal academic or news text.

Methods: The MTF-S2S Baseline

The authors propose MTF-S2S (Multi-task Field-shared Sequence to Sequence), a baseline model designed to leverage MATINF's multi-field structure. Unlike traditional MTL models that primarily share initial layers, MTF-S2S implements sharing at both the layer and module levels.

The module sharing architecture allows different tasks to share specific functional blocks while maintaining task-specific outputs.

Key technical features include:

  • Field-Sharing: The model uses shared encoders for question descriptions and shared encoder/decoder modules for the questions themselves.
  • Synchronized Training: To handle varying sub-dataset sizes, batch sizes are adjusted proportionally, ensuring balanced training progress across tasks.
  • Task-Specific Heads: Specialized fully connected layers handle the final output for distinct classification tasks, such as age-grouping versus topic categorization.

Results and Key Insights

Benchmarking with state-of-the-art models like BERT and ERNIE revealed significant advantages for the multi-task approach:

  • Performance Gains: The MTF-S2S model outperformed its single-task counterparts in both QA (+0.74 ROUGE-L) and summarization (+4.73 ROUGE-L). This suggests that learning to classify and summarize provides auxiliary signals that improve answer generation.
  • Efficiency: Testing with BERT-of-Theseus (a compressed 6-layer model) showed that high performance can be maintained with significantly reduced computational overhead compared to the full 12-layer BERT.
  • Domain Hardness: The fine-grained nature of the maternity domain proved difficult for traditional statistical methods, highlighting the need for deep semantic understanding.

Conclusion

MATINF addresses a critical need in the NLP community for cross-task datasets. By providing a large-scale, jointly labeled corpus, it enables more rigorous research into multi-task learning and general language representation. While the web-crawled nature of the data introduces some noise, MATINF serves as a robust playground for developing models that can handle the informal, domain-specific language typical of social media.

Summary updated

Abstract

Recently, large-scale datasets have vastly facilitated the development in nearly all domains of Natural Language Processing. However, there is currently no cross-task dataset in NLP, which hinders the development of multi-task learning. We propose MATINF, the first jointly labeled large-scale dataset for classification, question answering and summarization. MATINF contains 1.07 million question-answer pairs with human-labeled categories and user-generated question descriptions. Based on such rich information, MATINF is applicable for three major NLP tasks, including classification, question answering, and summarization. We benchmark existing methods and a novel multi-task baseline over MATINF to inspire further research. Our comprehensive comparison and experiments over MATINF and other datasets demonstrate the merits held by MATINF.
BibTeX
@inproceedings{xu2020matinf,
  author = {Canwen Xu and Jiaxin Pei and Hongtao Wu and Yiyu Liu and Chenliang Li},
  title = {{MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=Sgpmxm_iR4W},
}
Figure