Abstract
Aspect-Based Sentiment Classification (ABSC) requires a fine-grained understanding of specific entities within a text. While Large Language Models (LLMs) excel at general tasks, they often struggle with the precise contextual dependencies of ABSC. This paper presents a novel two-step "model editing" framework that identifies sentiment-specific neurons through causal tracing and performs surgical updates. By targeting mid-layers and aspect-word positions, the proposed method achieves state-of-the-art performance using only 0.006% of the trainable parameters required by full models, demonstrating extreme efficiency and robust generalization.
Introduction
Traditional sentiment analysis often categorizes an entire text as positive or negative. However, Aspect-Based Sentiment Classification (ABSC) demands a more nuanced approach, identifying the sentiment associated with specific "aspects" (e.g., "the food was great, but the service was slow").
Current approaches using Large Language Models (LLMs) typically rely on full fine-tuning or Parameter-Efficient Fine-Tuning (PEFT) like LoRA. These methods often update parameters across the entire architecture, which can be computationally expensive and introduce "noise" that leads to catastrophic forgetting. This research investigates whether specific "sentiment-specific" regions exist within LLMs and if targeting these regions through model editing can improve both efficiency and accuracy.
Methodology
The researchers propose a two-step surgical framework: Causal Tracing followed by Model Editing.
Causal Tracing Analysis
To locate where sentiment knowledge is stored, a "corrupt-and-restore" technique is employed. The model's input embeddings are intentionally corrupted with noise to break its prediction. Individual layers or tokens are then systematically restored to their clean state. A component is deemed "causally important" if its restoration significantly improves the model's ability to recover the correct sentiment prediction.
Model Editing
Once critical components are identified, the model is edited using two primary techniques:
- Weight-Based Editing: Low-rank updates are applied exclusively to the attention output projection matrices of the causally significant layers.
- Representation-Based Editing: The hidden states (residual stream) are manipulated at the specific positions of aspect terms to steer the model's output.
Experimental Setup
The framework was evaluated on Llama-2-7b across four diverse domains:
- Restaurant & Laptop: Standard benchmarks from SemEval 2014/2015/2016.
- Device: Digital device reviews.
- Service: Web service reviews.
Key Insights: The Mid-Layer and Aspect Anchor
Causal tracing revealed a distinct pattern in how LLMs process sentiment for ABSC. Unlike many tasks where the final token is most influential, ABSC relies heavily on the hidden states of the aspect words themselves.
As shown in the heatmap, the "sweet spot" for sentiment information in Llama-2-7b resides in the mid-layers (10–15). Editing these layers at the aspect token position proved significantly more effective than editing the last token or random positions.
Results
The surgical editing approach demonstrated several advantages over traditional PEFT methods:
- Accuracy: The method achieved an average in-domain accuracy of 87.6%, surpassing LoRA (86.5%) and LoReft (85.9%).
- Efficiency: By focusing only on sentiment-specific layers, the method utilized only 0.006% of the model's parameters, compared to 0.826% for LoRA.
- Generalization: The identified "sentiment-specific" neurons appeared consistent across domains, leading to more stable performance in out-of-domain testing.
Conclusion
This work demonstrates that LLMs possess localized, task-specific regions that can be targeted for highly efficient adaptation. By focusing on the mid-layers and aspect-token positions, model editing provides a "surgical" alternative to broad fine-tuning. Future work will explore whether these sentiment zones are consistent across different model architectures like Mistral or Falcon and how these techniques might be used to mitigate model bias.
Summary updated