TraceNet: Tracing and Locating the Key Elements in Sentiment Analysis

Qinghua Zhao, Shuai MaarXiv 2022 · OpenReview

Abstract

In this paper, we study sentiment analysis tasks where outcomes are primarily driven by a few key elements of the input. Motivated by the two-streams hypothesis of human perception, we propose a neural architecture named TraceNet. It not only learns discriminative representations via its encoders but also traces key elements simultaneously via its locators. Organized in a layer-wise manner, TraceNet employs smoothness regularization between adjacent encoder-locator combinations (ELCs), enforces sparsity constraints on locators, and utilizes a proactive masking strategy based on item weights. Experimental results demonstrate TraceNet's effectiveness in sentiment classification, robustness to adversarial attacks, and enhanced interpretability.

Introduction

In sentiment analysis (SA), a few key words often carry the weight of an entire sentence—terms like "masterpiece" or "disaster" are decisive. While standard attention mechanisms attempt to capture this, they often produce "dense" weights, assigning importance to almost every word. This lack of focus makes models brittle to minor perturbations and difficult for humans to interpret. TraceNet addresses these challenges by explicitly separating the "what" (semantic meaning) from the "where" (sentiment drivers), mimicking the brain's ventral and dorsal pathways.

Methodology

TraceNet is built upon Encoder-Locator Combinations (ELCs), which are stacked to iteratively refine the model's focus.

The overall architecture of TraceNet, featuring stacked Encoder-Locator Combinations (ELCs) and a proactive masking training strategy.

Key Components:

  1. Two-Stream Processing: The Encoder learns discriminative representations, while the Locator identifies key elements. This explicit separation allows for a clearer map of input structure.
  2. Layer-wise Sparsity: As data progresses through the ELC stack, the Locator faces increasingly strict sparsity constraints. This forces the model to "zoom in" on the most decisive subset of words.
  3. Proactive Masking: During training, the model identifies high-weight words and randomly masks them. This prevents over-reliance on a single "smoking gun" word, forcing the model to learn from broader context and increasing its resilience to noise.
  4. Smoothness Regularization: Adjacent weight tying ensures stable learning by preventing drastic changes in hidden states between layers.

Experimental Results

TraceNet was evaluated on the SST-5 and YELP-5 benchmarks using GloVe, XLNet, and RoBERTa backbones.

Performance and Robustness

TraceNet-R (RoBERTa-based) achieved accuracies of 57.34% on SST-5 and 67.70% on YELP-5, consistently outperforming base models. More importantly, it demonstrated superior adversarial resilience. In "Concatenation" attacks, TraceNet-X outperformed XLNet by 2.40%, proving that its focus on key elements makes it harder to fool with irrelevant noise.

Interpretability

Qualitative analysis via heatmaps reveals that TraceNet successfully ignores filler words to focus on sentiment-rich terms. As shown in the figure below, the model's weights become more refined and sparse in deeper layers (L1 to L3), effectively "tracing" the sentiment to specific descriptors.

Heatmap visualization of item weights across layers for the sentence "By turns very dark and very funny," showing increasing sparsity and focus in deeper layers.

Conclusion and Future Work

TraceNet offers a robust and interpretable framework for sentiment analysis by explicitly modeling the sparse structure of sentiment-bearing elements. While the proactive masking probability ($P_{msk}$) requires careful tuning to balance information loss and robustness, the architecture's success suggests broad applicability. Future work will explore applying this "trace and locate" logic to aspect-level sentiment analysis to handle more complex, multi-faceted inputs.

Summary updated

Abstract

In this paper, we study sentiment analysis task where the outcomes are mainly contributed by a few key elements of the inputs. Motivated by the two-streams hypothesis, we propose a neural architecture, named TraceNet, to address this type of task. It not only learns discriminative representations for the target task via its encoders, but also traces key elements at the same time via its locators. In TraceNet, both encoders and locators are organized in a layer-wise manner, and a smoothness regularization is employed between adjacent encoder-locator combinations. Moreover, a sparsity constraints are enforced on locators for tracing purposes and items are proactively masked according to the item weights output by locators.A major advantage of TraceNet is that the outcomes are easier to understand, since the most responsible parts of inputs are identified. Also, under the guidance of locators, it is more robust to attacks due to its focus on key elements and the proactive masking training strategy. Experimental results show its effectiveness for sentiment classification. Moreover, we provide several case studies to demonstrate its robustness and interpretability.
BibTeX
@misc{zhao2022tracenet,
  author = {Qinghua Zhao and Shuai Ma},
  title = {{TraceNet: Tracing and Locating the Key Elements in Sentiment Analysis}},
  year = {2022},
  eprint = {2202.13812},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.13812},
}
Figure