Abstract
This paper introduces HCapsNet, a novel capsule network architecture designed to address the challenge of polysemy in Natural Language Processing. By projecting word representations onto trainable hyperplanes, the model effectively disentangles the multiple semantic senses of a word. Experimental results across six major text classification benchmarks demonstrate that HCapsNet outperforms standard CNNs, LSTMs, and vanilla capsule networks on five datasets. Visual analysis using t-SNE further confirms that the hyperplane projection mechanism successfully clusters relevant semantic features, providing a more robust representation for classification tasks.
Introduction: The Polysemy Problem
In modern NLP, words are typically represented as dense vectors or embeddings. However, traditional techniques like Word2Vec or GloVe often suffer from the "conflation" problem: they compress all meanings of a polysemic word into a single vector. For instance, the word "cold" can refer to temperature, a viral infection, or a person's demeanor. A single vector representation acts as a weighted average of these meanings, introducing ambiguity that can hinder classification accuracy.
While contextual models like BERT adjust vectors based on surrounding words, the authors of HCapsNet argue for a more explicit structural approach. They build on the theory that multiple senses of a word reside in a linear superposition within the embedding space and propose a geometric solution to "slice" out specific semantic features.
HCapsNet Architecture and Methods
HCapsNet consists of three primary layers designed to capture and refine semantic information:
- Bi-directional RNN Layer: This layer captures long-distance dependencies and provides initial contextualized representations for each word.
- Convolutional Capsule Layer: This layer groups detected features into "primary capsules," which represent basic semantic units.
- Fully Connected Capsule Layer: This is the core innovation of the model, utilizing a mechanism called Routing-on-Hyperplane.
Routing-on-Hyperplane
Instead of passing the entire capsule vector to the next layer, the model attaches a trainable hyperplane to each capsule. The process involves:
- Projection: The capsule's output vector $u_i$ is projected onto a hyperplane defined by a normal vector $w_i$. The resulting vector $u_{\perp i}$ effectively filters the input to extract a specific semantic "sense."
- Dynamic Routing: An iterative process determines which projected senses agree most with high-level category capsules (e.g., "Positive Sentiment").
- Diversity Penalization: To ensure the model captures a wide range of meanings, a penalization term encourages the hyperplanes' normal vectors to be orthogonal, preventing redundant projections.
Key Results and Visual Proof
HCapsNet achieved state-of-the-art performance on 5 out of 6 evaluated datasets, including SST-2 (88.7% accuracy) and AG’s News (93.5%). The model's success is attributed to its ability to preserve the "pose" (direction and meaning) of features through vector-based capsules rather than scalar neurons.
The authors provided visual evidence of this semantic disentanglement using t-SNE. As shown in the visualizations above, word vectors are often scattered and ambiguous before projection. However, after the hyperplane projection, words sharing the same contextual sense cluster tightly together. For example, a case study on the word "wonder" showed that the model could distinguish between its use in a positive movie review ("wonder tales") versus a neutral, inquisitive context ("you begin to wonder").
Conclusion and Future Work
HCapsNet demonstrates that treating polysemy as a geometric projection problem significantly enhances text classification. While the model showed less improvement on keyword-heavy tasks like the TREC question dataset, its ability to handle subtle semantic shifts makes it a powerful tool for sentiment and topic analysis. Future research may extend this hyperplane decomposition to more complex tasks such as Machine Translation and Reading Comprehension, where precise word-sense disambiguation is critical.
Summary updated