MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification

Ye Zhang, Stephen Roller, Byron C. Wallace

Introduction

Word embeddings are fundamental to modern Natural Language Processing (NLP). Different embedding techniques capture distinct linguistic features: word2vec often captures broad semantic associations from large news corpora, while dependency-based embeddings focus on syntactic roles. A persistent challenge in sentence classification is effectively combining these diverse "views" of language.

Previous methods, such as the Multi-View CNN (MVCNN), treated multiple embeddings as separate channels, similar to RGB channels in an image. This approach imposes rigid constraints, requiring all input embeddings to share the same dimensionality. Furthermore, these models often involve computationally intensive pre-training or mutual learning phases. The Multi-Group Norm Constraint CNN (MGNC-CNN) addresses these limitations by providing a flexible, efficient, and high-performing architecture for leveraging multiple word embedding types simultaneously.

Methodology

The authors propose a modular architecture that treats different embedding sets as independent groups rather than aligned channels.

MG-CNN (Multi-Group CNN)

In the MG-CNN variant, each set of embeddings is processed by its own independent convolutional layer. The resulting feature vectors are concatenated at the penultimate layer before being passed to a softmax classifier. This design allows for the integration of embeddings with varying dimensions (e.g., 300d word2vec and 100d GloVe) without requiring padding or linear projections.

MGNC-CNN (Multi-Group Norm Constraint CNN)

The MGNC-CNN extends this architecture by introducing a specialized regularization strategy. While standard CNNs often apply a single $l2$ norm constraint to the final feature vector, MGNC-CNN applies separate constraints ($\lambda_l$) to the sub-vectors generated by each embedding group.

Architecture of the MGNC-CNN model showing the flow from multiple embeddings to the final softmax classification.

By tuning these group-specific $\lambda$ values on a validation set, the model learns to dynamically weight the contributions of different embedding types. This allows the network to prioritize more informative features while penalizing noise from less relevant embedding sets for a given task.

Experimental Setup

The model was evaluated across five benchmark datasets for sentence classification:

  • SST-1 & SST-2: Stanford Sentiment Treebank (fine-grained and binary).
  • Subj: Subjectivity classification.
  • TREC: Question type categorization.
  • Irony: Detection of ironic Reddit comments.

The study utilized three pre-trained embedding types: word2vec (trained on Google News), GloVe (trained on Common Crawl), and custom syntactic embeddings derived from dependency parses.

Results and Discussion

The MGNC-CNN demonstrates significant improvements over single-embedding baselines and simple concatenation methods.

Performance comparison of MGNC-CNN against various baselines across five benchmark datasets.

Key findings include:

  • State-of-the-Art Performance: The model achieved top-tier results on the Subj and TREC datasets (95.5% accuracy on TREC). On the Irony dataset, it provided a 4% improvement in AUC over the baseline.
  • Efficiency: Unlike complex multi-view models that can take days to train, MGNC-CNN typically converges in approximately one hour.
  • Flexibility: The "multi-group" approach is more effective for text than the "multi-channel" approach because linguistic features in different embedding spaces are not spatially aligned like image pixels.

Conclusion

MGNC-CNN offers a simple yet robust framework for exploiting multiple word embeddings. By treating embeddings as independent groups and applying separate norm constraints, the model achieves high performance with minimal computational overhead. While the approach introduces additional hyperparameters (the $\lambda$ values for each group), the tuning process is highly parallelizable. Future research may explore automated methods for discovering these regularization constraints or extending the multi-group architecture to other linguistic features.

Summary updated

BibTeX
@inproceedings{zhang2016mgnc,
  author = {Ye Zhang and Stephen Roller and Byron C. Wallace},
  title = {{MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=rkZGTzWdbB},
}
Figure