MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization

Canwen Xu, Jiaxin Pei, Hongtao Wu, Yiyu Liu, Chenliang Li

Abstract

Recently, large-scale datasets have vastly facilitated the development in nearly all domains of Natural Language Processing. However, there is currently no cross-task dataset in NLP, which hinders the development of multi-task learning. We propose MATINF, the first jointly labeled large-scale dataset for classification, question answering and summarization. MATINF contains 1.07 million question-answer pairs with human-labeled categories and user-generated question descriptions. Based on such rich information, MATINF is applicable for three major NLP tasks, including classification, question answering, and summarization. We benchmark existing methods and a novel multi-task baseline over MATINF to inspire further research. Our comprehensive comparison and experiments over MATINF and other datasets demonstrate the merits held by MATINF.

1. Introduction

In Natural Language Processing (NLP), the availability of large-scale datasets like SQuAD for question answering or CNN/Daily Mail for summarization has driven significant progress. However, these datasets are typically "single-task," providing supervision for only one specific objective. This limitation poses a challenge for Multi-Task Learning (MTL). When a model's performance improves through training on multiple datasets, it is often difficult to determine whether the gain stems from the increased volume of input data or the diversity of supervision signals.

To address this, researchers from Wuhan University and the University of Michigan have introduced MATINF, a large-scale Chinese dataset jointly labeled for three distinct tasks: text classification, question answering (QA), and summarization. By providing multiple labels for the same input text, MATINF serves as a "fair playground" for evaluating MTL strategies.

2. The MATINF Dataset

MATINF is sourced from a prominent Chinese maternity and childcare forum. It comprises over 1.07 million unique entries, each containing:

  • Question (Q): A concise query.
  • Description (D): A detailed explanation of the user's situation.
  • Category (C): Human-labeled metadata (18 topics and 3 age groups).
  • Best Answer (A): A crowd-voted response.

This rich structure supports three primary tasks:

  1. Classification: Predicting the category $C$ given $Q$ and $D$.
  2. Question Answering: Generating the answer $A$ given $Q$.
  3. Summarization: Generating the concise $Q$ from the detailed $D$.

Unlike general news datasets, MATINF focuses on a fine-grained professional domain, requiring models to distinguish between closely related topics like "infant nutrition" and "infant feeding."

3. Multi-Task Learning Paradigms

The core advantage of MATINF is its ability to isolate the effects of multi-task supervision. In traditional MTL, models are often trained on disparate datasets with different inputs ($X_1, X_2$) for different tasks ($Y_1, Y_2$). MATINF allows for a unified approach where the same input $X$ is mapped to multiple outputs.

Comparison of traditional multi-task learning vs. MATINF and different sharing strategies.

As illustrated in the figure above, MATINF enables a shift from "Traditional" MTL (a) to a more integrated paradigm (b). Furthermore, the dataset allows researchers to explore different architectural sharing strategies, moving from simple "Layer sharing" (c) to more complex "Module sharing" (d).

4. MTF-S2S: A Multi-task Baseline

The authors propose MTF-S2S (Multi-task Field-shared Sequence to Sequence) as a baseline. Unlike models that only share lower-level layers, MTF-S2S utilizes shared modules, including a shared LSTM encoder for questions and descriptions and a shared decoder for answers. Task-specific heads are employed for classification tasks, while the core representation logic is shared across all objectives. To balance training across tasks of varying scales, the authors implemented a synchronized training regimen using a batch-size balancing formula.

5. Experimental Results

Benchmarking on MATINF revealed several key insights:

  • Classification: Pre-trained models like BERT and ERNIE achieved the highest accuracy (~66.9%). Notably, the multi-task version of the MTF-S2S model outperformed its single-task counterpart, validating the benefits of joint training.
  • Question Answering: MTF-S2S reached a ROUGE-L of 14.26. While it outperformed standard Seq2Seq models, the gap between current performance and the "Best Passage" upper bound (49.00) indicates significant room for future research in generative QA.
  • Summarization: MATINF-SUMM proved more challenging than news-based datasets like LCSTS. The BERT-based BertAbs model performed best (ROUGE-L 55.93), but the authors' multi-task model showed a substantial improvement (+4.73 ROUGE-L) over the single-task baseline.

6. Conclusion

MATINF provides a unique resource for the NLP community, offering a large-scale, domain-specific dataset with joint labels. By eliminating input bias, it allows for a rigorous evaluation of multi-task learning architectures. While the dataset contains real-world noise from user-generated content, this characteristic also serves as a test for model robustness. Future work will likely leverage MATINF to develop more sophisticated MTL models and explore how common knowledge is represented across diverse linguistic tasks.

Summary updated

Abstract

Recently, large-scale datasets have vastly facilitated the development in nearly all domains of Natural Language Processing. However, there is currently no cross-task dataset in NLP, which hinders the development of multi-task learning. We propose MATINF, the first jointly labeled large-scale dataset for classification, question answering and summarization. MATINF contains 1.07 million question-answer pairs with human-labeled categories and user-generated question descriptions. Based on such rich information, MATINF is applicable for three major NLP tasks, including classification, question answering, and summarization. We benchmark existing methods and a novel multi-task baseline over MATINF to inspire further research. Our comprehensive comparison and experiments over MATINF and other datasets demonstrate the merits held by MATINF.
BibTeX
@inproceedings{xu2020matinf,
  author = {Canwen Xu and Jiaxin Pei and Hongtao Wu and Yiyu Liu and Chenliang Li},
  title = {{MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=Sgpmxm_iR4W},
}
Figure