Abstract: Capturing the compositional process which maps the meaning of words to that of documents is a central challenge for researchers in Natural Language Processing and Information Retrieval. We introduce a model that is able to represent the meaning of documents by embedding them in a low dimensional vector space, while preserving distinctions of word and sentence order crucial for capturing nuanced semantics. Our model is based on an extended Dynamic Convolution Neural Network, which learns convolution filters at both the sentence and document level, hierarchically learning to capture and compose low level lexical features into high level semantic concepts. We demonstrate the effectiveness of this model on a range of document modelling tasks, achieving strong results with no feature engineering and with a more compact model. Inspired by recent advances in visualising deep convolution networks for computer vision, we present a novel visualisation technique for our document networks which not only provides insight into their learning process, but also can be interpreted to produce a compelling automatic summarisation system for texts.
Introduction
A central challenge in Natural Language Processing (NLP) is compositionality: the mathematical representation of a document's meaning based on its constituent words. Traditional "bag-of-words" models often fail to capture nuanced semantics because they ignore word order and structural hierarchy. While Recursive Neural Networks have attempted to address this by following tree structures, they often require expensive and potentially inaccurate syntactic parsing.
This research introduces a hierarchical approach using Convolutional Neural Networks (ConvNets) to bridge the gap between Computer Vision (CV) and NLP. By adapting CV techniques, the authors demonstrate that a single architecture can perform sentiment classification while simultaneously identifying the most salient parts of a text for automatic summarization.
Hierarchical Architecture
The core of the proposed model is an extended Dynamic Convolutional Neural Network (DCNN) that mirrors the natural hierarchy of text (words $\rightarrow$ sentences $\rightarrow$ documents). The architecture operates at two distinct levels:
- Sentence Level: The model takes word embeddings and applies wide convolutions followed by $k$-max pooling. This process creates a fixed-length "sentence embedding" that captures local semantic features while remaining invariant to the specific position of words.
- Document Level: These sentence embeddings are then treated as the input units for a second ConvNet. This higher-level network composes sentence representations into a single vector representing the entire document.
A key innovation in this design is treating word embeddings as "channels," similar to RGB channels in an image. This approach significantly reduces the number of parameters compared to previous text-based ConvNets, resulting in a more compact and efficient model.
Visualisation and Saliency
To understand the model's decision-making process, the authors utilize saliency mapping. By calculating the gradient of the output with respect to the input, the model identifies which words and sentences most heavily influence its classification. The authors discovered that the sentences the model finds most "salient" for classification are typically the most descriptive. This allows the model to function as an extractive summarizer by simply selecting the top $k$ salient sentences, without requiring explicit training for the summarization task.
Experimental Results
The model was evaluated on two primary benchmarks: the Twitter Sentiment Dataset (short, noisy text) and the IMDB Movie Review Dataset (long-form documents).
On the Twitter dataset, the model achieved 46 errors, nearly matching the state-of-the-art DCNN while maintaining a smaller parameter footprint. On the IMDB dataset, it reached 89.38% accuracy, placing it among the top-performing models at the time of publication.
The summarization capabilities were particularly robust. In a secondary evaluation, a classifier was trained on summaries generated by the model. As shown in the results, even when a review was stripped down to just 20% of its original length using the most salient sentences, classification accuracy dropped by less than 1%. This significantly outperformed common heuristics, such as selecting the first and last sentences of a document.
Conclusion
By forcing the model to represent sentences as intermediate "bottleneck" vectors, the authors created a system that generalizes well across different document lengths and types. The resulting architecture is not only a powerful tool for sentiment analysis but also a highly effective, unsupervised extractive summarizer that preserves the crucial semantic order of natural language.
Summary updated