A unified architecture for natural language processing: deep neural networks with multitask learning

Ronan Collobert, Jason WestonICML 2008 · OpenReview

Abstract

We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance.
BibTeX
@inproceedings{collobert2008unified,
  author = {Ronan Collobert and Jason Weston},
  title = {{A unified architecture for natural language processing: deep neural networks with multitask learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2008},
  url = {https://openreview.net/forum?id=HJWhcjWOWS},
}

Loading related content...

Figure