This research, published in 2017, introduces a "divide-and-conquer" approach to text classification that challenges the then-dominant trend of building increasingly deep hierarchical models. The proposed Multi-View Network (MVN) emphasizes network width over depth, utilizing a "team of readers" to capture diverse linguistic nuances.
Problem Statement
At the time of publication, state-of-the-art text classification relied heavily on deep hierarchical models like LSTMs or deep CNNs. These models typically act as a single reader processing text linearly or through layers of increasing abstraction. However, deep models often struggle with gradient flow—making them difficult to train—and can be overly sensitive to noise. The authors argue that a single "view" of a sentence might miss subtle nuances, proposing instead that multiple specialized readers (views) should pool their knowledge to make a final decision.
Methodology
The MVN architecture consists of three main stages designed to facilitate diverse feature extraction and efficient training:
- Multiple Attentions for Selection: The model creates $V$ different "selection vectors." Each vector is a weighted sum of the input words, where the weights are determined by a view-specific softmax attention mechanism. This allows different views to focus on different keywords (e.g., one view might focus on adjectives for sentiment, while another focuses on nouns for topic).
- Aggregating Selections into Views: This is the core novelty of the architecture. The views are arranged sequentially. While the first view is an independent attempt at feature selection, subsequent views are "stacked." Each intermediate view is a function of the current selection plus a concatenation of all previous views. This creates a "short-cutting" effect similar to Residual Networks, allowing later views to either build upon or deviate from earlier ones.
- Classification: Finally, all views are concatenated and passed through a two-layer perceptron with a softmax output to predict the class.
The model is highly flexible; while it performs well with simple bag-of-words embeddings, it can be augmented with convolutional features (max-pooled n-grams) to capture local context.
Key Insights and Ideas
Width Over Depth
By expanding the network "horizontally" with more views rather than "vertically" with more layers, the authors created shorter connections between the input and the loss function. This significantly improves gradient flow and speeds up convergence compared to deeper alternatives.
View Specialization
Through post-hoc analysis, the authors discovered that views actually specialize. For example, in sentiment analysis, some views were significantly better at identifying "neutral" sentences, while others excelled at "very negative" ones. This confirms the "team of readers" hypothesis where different components of the model learn to handle different aspects of the data.
Sequential Synergy
Ablation studies demonstrated that "horizontal links"—allowing views to see previous views—are vital. A model where views were independent performed worse than the full MVN, proving that the "team" performs better when members can build on each other's work.
Experimental Results
The MVN established new state-of-the-art (SOTA) performance on major benchmarks at the time of its release:
- Stanford Sentiment Treebank (SST): The model achieved 51.5% accuracy on fine-grained (5-class) sentiment prediction, surpassing high-order CNNs (51.2%) and Tree-LSTMs (51.0%).
- AG English News Corpus: On this large-scale dataset, the MVN achieved a 7.13% error rate, outperforming very deep 29-layer CNNs (8.67%) and traditional n-gram approaches.
- Efficiency: The architecture proved highly efficient, reaching its best results on the AG News validation set in just a few thousand iterations.
Conclusion and Future Work
While the MVN offers significant advantages in training efficiency and performance, the authors note that parameter counts grow quadratically with the number of views, which may limit the model's width for extremely complex tasks. Future work includes exploring even more diverse base features, such as syntactic dependencies, and expanding the multi-view philosophy to tasks like machine translation and sequence tagging.
Summary updated