Abstract
In this paper, we study sentiment analysis tasks where outcomes are primarily driven by a few key elements of the input. Motivated by the two-streams hypothesis of human perception, we propose a neural architecture named TraceNet. It not only learns discriminative representations via its encoders but also traces key elements simultaneously via its locators. Organized in a layer-wise manner, TraceNet employs smoothness regularization between adjacent encoder-locator combinations (ELCs), enforces sparsity constraints on locators, and utilizes a proactive masking strategy based on item weights. Experimental results demonstrate TraceNet's effectiveness in sentiment classification, robustness to adversarial attacks, and enhanced interpretability.
Introduction
In sentiment analysis (SA), a few key words often carry the weight of an entire sentence—terms like "masterpiece" or "disaster" are decisive. While standard attention mechanisms attempt to capture this, they often produce "dense" weights, assigning importance to almost every word. This lack of focus makes models brittle to minor perturbations and difficult for humans to interpret. TraceNet addresses these challenges by explicitly separating the "what" (semantic meaning) from the "where" (sentiment drivers), mimicking the brain's ventral and dorsal pathways.
Methodology
TraceNet is built upon Encoder-Locator Combinations (ELCs), which are stacked to iteratively refine the model's focus.
Key Components:
- Two-Stream Processing: The Encoder learns discriminative representations, while the Locator identifies key elements. This explicit separation allows for a clearer map of input structure.
- Layer-wise Sparsity: As data progresses through the ELC stack, the Locator faces increasingly strict sparsity constraints. This forces the model to "zoom in" on the most decisive subset of words.
- Proactive Masking: During training, the model identifies high-weight words and randomly masks them. This prevents over-reliance on a single "smoking gun" word, forcing the model to learn from broader context and increasing its resilience to noise.
- Smoothness Regularization: Adjacent weight tying ensures stable learning by preventing drastic changes in hidden states between layers.
Experimental Results
TraceNet was evaluated on the SST-5 and YELP-5 benchmarks using GloVe, XLNet, and RoBERTa backbones.
Performance and Robustness
TraceNet-R (RoBERTa-based) achieved accuracies of 57.34% on SST-5 and 67.70% on YELP-5, consistently outperforming base models. More importantly, it demonstrated superior adversarial resilience. In "Concatenation" attacks, TraceNet-X outperformed XLNet by 2.40%, proving that its focus on key elements makes it harder to fool with irrelevant noise.
Interpretability
Qualitative analysis via heatmaps reveals that TraceNet successfully ignores filler words to focus on sentiment-rich terms. As shown in the figure below, the model's weights become more refined and sparse in deeper layers (L1 to L3), effectively "tracing" the sentiment to specific descriptors.
Conclusion and Future Work
TraceNet offers a robust and interpretable framework for sentiment analysis by explicitly modeling the sparse structure of sentiment-bearing elements. While the proactive masking probability ($P_{msk}$) requires careful tuning to balance information loss and robustness, the architecture's success suggests broad applicability. Future work will explore applying this "trace and locate" logic to aspect-level sentiment analysis to handle more complex, multi-faceted inputs.
Summary updated