Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity

Po-Nien Kung, Sheng-Siang Yin, Yi-Cheng Chen, Tse-Hsuan Yang, Yun-Nung ChenEMNLP 2021 · OpenReview

Abstract: Po-Nien Kung, Sheng-Siang Yin, Yi-Cheng Chen, Tse-Hsuan Yang, Yun-Nung Chen. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

2021.

Introduction

Multi-task learning (MTL) is a powerful paradigm where a primary task is trained alongside auxiliary tasks to improve generalization. However, MTL faces two significant hurdles: task selection and computational cost. Determining which auxiliary tasks provide positive transfer is non-trivial, as unrelated data can introduce noise. Furthermore, training on massive auxiliary datasets is resource-intensive. Existing automated selection methods often rely on complex reinforcement learning loops that are themselves computationally expensive. This paper proposes an efficient two-stage pipeline that selects specific, high-similarity data samples from auxiliary tasks to optimize primary task performance.

Methods: The TO-MTDNN Pipeline

The authors introduce the Task-Oriented Multi-Task Deep Neural Network (TO-MTDNN), a framework designed to select only the most relevant auxiliary data samples based on feature similarity.

The two-stage TO-MTDNN pipeline involving task-discriminative training, similarity ranking, and task-oriented fine-tuning.

Stage 1: Task-Discriminative Training & Ranking

In the first stage, a "toy" model (TD-MTDNN) is trained on a minimal subset of data (500 samples per task). A task discriminator head is added to the model to predict the task origin of an input based on its last hidden state. Once trained, this discriminator scores the remaining auxiliary data. Samples that the discriminator identifies as similar to the primary task are prioritized. The authors found that using Binary Cross Entropy (BCE) for the discriminator outperformed standard Cross Entropy (CE) because BCE allows samples to share similarities across multiple tasks simultaneously, rather than forcing a single-task classification.

Stage 2: Task-Oriented Training & Fine-Tuning

In the second stage, the top-$N$ ranked auxiliary samples are combined with the full primary dataset to train a fresh model (TO-MTDNN). Finally, the model is fine-tuned exclusively on the primary task to achieve peak performance.

Experimental Setup

The method was evaluated on the GLUE (General Language Understanding Evaluation) benchmark, focusing on:

  • Primary Tasks: RTE (Recognizing Textual Entailment), MRPC (Microsoft Research Paraphrase Corpus), and STS-B (Semantic Textual Similarity Benchmark).
  • Auxiliary Tasks: MNLI, QQP, QNLI, SST-2, and CoLA.

Key Insights and Ideas

A central finding is that the granularity of selection matters; selecting specific samples is more effective than selecting entire tasks. The model's internal representations (hidden states) serve as effective proxies for task relatedness.

t-SNE visualizations comparing feature distributions in the toy model, BCE-based discriminator, and CE-based discriminator.

The choice of loss function in Stage 1 is critical. As seen in the t-SNE visualizations, the BCE loss allows for more nuanced, overlapping task representations compared to the "winner-take-all" nature of CE. This flexibility better captures the linguistic overlaps between different NLP tasks, leading to more effective data selection.

Key Results

The TO-MTDNN approach consistently outperformed single-task models and standard MT-DNNs using random sampling. Notably, the method achieved superior performance while using significantly less auxiliary data:

  • RTE: 50% of auxiliary data.
  • MRPC: 66% of auxiliary data.
  • STS-B: Only 1% of auxiliary data.

The pipeline also offers a massive speedup, running 12-15x faster than a full MT-DNN. For instance, on the RTE task, the proposed method reduced training time from approximately 4.4 hours to under 20 minutes. The method also demonstrated high robustness in low-resource (few-shot) scenarios, where the primary task has very little data.

Limitations and Future Work

While effective, the initial sample size for the toy model (500 samples) is currently a heuristic that could be further optimized. Additionally, the current pipeline focuses on a single primary task; extending it to support multiple primary tasks simultaneously is a promising direction for future research. Finally, testing the "feature similarity" selection logic in other domains like Computer Vision or Speech Recognition would validate its generalizability beyond NLP.

Summary updated

BibTeX
@inproceedings{kung2021efficient,
  author = {Po-Nien Kung and Sheng-Siang Yin and Yi-Cheng Chen and Tse-Hsuan Yang and Yun-Nung Chen},
  title = {{Efficient Multi-Task Auxiliary Learning: Selecting Auxiliary Data by Feature Similarity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=zKUNNWQUy8I},
}
Figure