Research on Task Discovery for Transfer Learning in Deep Neural Networks

Arda AkdemirACL 2020

TL;DR

This paper explores how to make deep learning models more like humans by learning from multiple tasks simultaneously. The authors propose novel approaches to optimize model architectures, discover beneficial auxiliary tasks, and develop smarter pretraining strategies to improve generalization and avoid "negative transfer" in various machine learning domains.

What Problem Does This Paper Address?

Deep neural networks, while powerful, often struggle with generalization to new, unseen data. They typically require vast amounts of labeled data for each new task and are trained from scratch, a stark contrast to how humans leverage existing knowledge to learn new skills. Transfer learning aims to bridge this gap by using knowledge gained from one task to improve performance on another. However, current transfer learning methods often fall short due to challenges in selecting the right tasks, determining how to share information between tasks, and the risk of "catastrophic forgetting" (where a model forgets previously learned information). This paper tackles these limitations by focusing on "task discovery" – finding optimal ways to combine tasks and share knowledge to unlock the full potential of transfer learning.

Methods

The research is structured around three key research questions (RQs):

  • RQ1: Optimizing Model Architecture and Sharing for Task Combinations: The authors propose using learnable parameters to control information flow between task-specific models, moving beyond simple hard-sharing. They also explore learning "joint label embeddings" to better capture relationships between disparate label classes, exemplified by combining Named Entity Recognition (NER) and Dependency Parsing.
  • RQ2: Identifying Beneficial Auxiliary Tasks: This involves fixing a target task and exploring how incorporating various auxiliary tasks, along with sophisticated data selection mechanisms, can improve performance. The hypothesis is that tasks like biological named entity recognition can aid biological question answering.
  • RQ3: Discovering Useful Pretraining Schemes: The paper proposes "adaptive masking" for pretraining language models. Unlike standard random masking, this method adaptively adjusts the difficulty of training instances based on model performance, aiming to improve the learning process by mimicking how humans learn from varying levels of challenge.

Datasets

The paper mentions the use of several datasets for training and evaluation:

  • NLP Tasks:
    • Turkish Named Entity Recognition and Dependency Parsing datasets (for RQ1).
  • PubMed abstracts (for RQ2).
  • SQuAD 2.0 (Stanford Question Answering Dataset) for downstream evaluation of pretraining (for RQ3).
  • Wikipedia articles (unlabeled) for pretraining (for RQ3).
  • Bioinformatics:
    • BC2GM, BC4CHEMD, and BC5CDR datasets for biological named entity recognition (for RQ2).
  • BioASQ dataset for biomedical question answering (for RQ2).
  • Computer Vision:
    • Office-Caltech dataset for multi-task object detection (for RQ2).
  • Caltech-256 dataset (for multi-task object detection).

Key Insights and Ideas

  • Human-like Learning: The core idea is to move beyond the "train from scratch" paradigm and emulate human learning by leveraging prior knowledge from related tasks.
  • Task Discovery is Crucial: Simply combining tasks isn't enough; understanding the relationships between them and how to share information is key to achieving significant improvements and avoiding negative transfer.
  • Beyond Hard-Sharing: Simple hard-sharing of low-level layers in multi-task learning can be suboptimal, especially for dissimilar tasks. More sophisticated methods like learnable parameter control and joint label embeddings are promising.
  • Adaptive Pretraining: Standard pretraining methods like BERT's masked language modeling might not be optimal for all downstream tasks. "Adaptive masking" offers a more dynamic and potentially more effective way to prepare models for specific tasks.
  • Cross-Domain Applicability: The research aims to extend these principles beyond NLP to domains like Bioinformatics and Computer Vision, highlighting the broad applicability of transfer learning.

Key Results

The paper presents preliminary results and outlines planned evaluations:

  • RQ1 (Turkish NER/DP): A proposed multi-task learning framework achieved an absolute 2.45% F-1 score improvement over a conventional joint learning model. A hierarchical multi-task learning framework further improved performance by 18.86% and 4.61% F-1 over their previous model for Dependency Parsing (DEP) and Named Entity Recognition (NER) respectively.
  • RQ2 (BioASQ Factoid QA): Initial results show improvements in Lenient Accuracy (LAcc), Strict Accuracy (SAcc), and Mean Reciprocal Rank (MRR) by incorporating transfer learning on PubMed abstracts and pretraining on SQuAD. The proposed MTL of all question types yielded the highest scores: LAcc 0.42, SAcc 0.61, MRR 0.49.
  • RQ3: Not explicitly detailed in terms of quantitative results yet, but the paper proposes "adaptive masking" as a novel pretraining scheme to be evaluated against BERT's standard pretraining.

Limitations and Future Work

  • Task-Relatedness: A significant challenge remains in thoroughly understanding and automatically identifying task-relatedness, which is crucial for effective transfer learning.
  • Architectural Variations: While task relationships are studied, more research is needed on how architectural choices interact with task sharing.
  • Generalizability: The paper acknowledges the need to test hypotheses on other less-resourced, morphologically rich languages (e.g., Czech) and across more diverse domains.
  • Data Selection: While promising, the effectiveness of various data selection mechanisms needs to be thoroughly evaluated across different task combinations.
  • Future Work: The authors plan to extend their models to tackle tasks from Bioinformatics and Computer Vision, specifically biomedical question answering and object detection. They also aim to further improve multi-task learning models by exploring novel methods.

Summary updated

BibTeX
@inproceedings{akdemir2020research,
  author = {Arda Akdemir},
  title = {{Research on Task Discovery for Transfer Learning in Deep Neural Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure