End-to-End Multi-View Networks for Text Classification

Hongyu Guo, Colin Cherry, Jiang SuarXiv 2017 · OpenReview

This research, published in 2017, introduces a "divide-and-conquer" approach to text classification that challenges the then-dominant trend of building increasingly deep hierarchical models. The proposed Multi-View Network (MVN) emphasizes network width over depth, utilizing a "team of readers" to capture diverse linguistic nuances.

Problem Statement

At the time of publication, state-of-the-art text classification relied heavily on deep hierarchical models like LSTMs or deep CNNs. These models typically act as a single reader processing text linearly or through layers of increasing abstraction. However, deep models often struggle with gradient flow—making them difficult to train—and can be overly sensitive to noise. The authors argue that a single "view" of a sentence might miss subtle nuances, proposing instead that multiple specialized readers (views) should pool their knowledge to make a final decision.

Methodology

The MVN architecture consists of three main stages designed to facilitate diverse feature extraction and efficient training:

  1. Multiple Attentions for Selection: The model creates $V$ different "selection vectors." Each vector is a weighted sum of the input words, where the weights are determined by a view-specific softmax attention mechanism. This allows different views to focus on different keywords (e.g., one view might focus on adjectives for sentiment, while another focuses on nouns for topic).
  2. Aggregating Selections into Views: This is the core novelty of the architecture. The views are arranged sequentially. While the first view is an independent attempt at feature selection, subsequent views are "stacked." Each intermediate view is a function of the current selection plus a concatenation of all previous views. This creates a "short-cutting" effect similar to Residual Networks, allowing later views to either build upon or deviate from earlier ones.
  3. Classification: Finally, all views are concatenated and passed through a two-layer perceptron with a softmax output to predict the class.
The Multi-View Network (MVN) architecture showing sequential views and horizontal links.

The model is highly flexible; while it performs well with simple bag-of-words embeddings, it can be augmented with convolutional features (max-pooled n-grams) to capture local context.

Key Insights and Ideas

Width Over Depth

By expanding the network "horizontally" with more views rather than "vertically" with more layers, the authors created shorter connections between the input and the loss function. This significantly improves gradient flow and speeds up convergence compared to deeper alternatives.

View Specialization

Through post-hoc analysis, the authors discovered that views actually specialize. For example, in sentiment analysis, some views were significantly better at identifying "neutral" sentences, while others excelled at "very negative" ones. This confirms the "team of readers" hypothesis where different components of the model learn to handle different aspects of the data.

F-measure across different class categories for individual views, demonstrating view specialization.

Sequential Synergy

Ablation studies demonstrated that "horizontal links"—allowing views to see previous views—are vital. A model where views were independent performed worse than the full MVN, proving that the "team" performs better when members can build on each other's work.

Experimental Results

The MVN established new state-of-the-art (SOTA) performance on major benchmarks at the time of its release:

  • Stanford Sentiment Treebank (SST): The model achieved 51.5% accuracy on fine-grained (5-class) sentiment prediction, surpassing high-order CNNs (51.2%) and Tree-LSTMs (51.0%).
  • AG English News Corpus: On this large-scale dataset, the MVN achieved a 7.13% error rate, outperforming very deep 29-layer CNNs (8.67%) and traditional n-gram approaches.
  • Efficiency: The architecture proved highly efficient, reaching its best results on the AG News validation set in just a few thousand iterations.

Conclusion and Future Work

While the MVN offers significant advantages in training efficiency and performance, the authors note that parameter counts grow quadratically with the number of views, which may limit the model's width for extremely complex tasks. Future work includes exploring even more diverse base features, such as syntactic dependencies, and expanding the multi-view philosophy to tasks like machine translation and sequence tagging.

Summary updated

Abstract

We propose a multi-view network for text classification. Our method automatically creates various views of its input text, each taking the form of soft attention weights that distribute the classifier's focus among a set of base features. For a bag-of-words representation, each view focuses on a different subset of the text's words. Aggregating many such views results in a more discriminative and robust representation. Through a novel architecture that both stacks and concatenates views, we produce a network that emphasizes both depth and width, allowing training to converge quickly. Using our multi-view architecture, we establish new state-of-the-art accuracies on two benchmark tasks.
BibTeX
@misc{guo2017end,
  author = {Hongyu Guo and Colin Cherry and Jiang Su},
  title = {{End-to-End Multi-View Networks for Text Classification}},
  year = {2017},
  eprint = {1704.05907},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.05907},
}
Figure