Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification

Shichen Li, Zhongqing Wang, Zheyu Zhao, Yue Zhang, Peifeng Li

Abstract

Aspect-Based Sentiment Classification (ABSC) requires a fine-grained understanding of specific entities within a text. While Large Language Models (LLMs) excel at general tasks, they often struggle with the precise contextual dependencies of ABSC. This paper presents a novel two-step "model editing" framework that identifies sentiment-specific neurons through causal tracing and performs surgical updates. By targeting mid-layers and aspect-word positions, the proposed method achieves state-of-the-art performance using only 0.006% of the trainable parameters required by full models, demonstrating extreme efficiency and robust generalization.

Introduction

Traditional sentiment analysis often categorizes an entire text as positive or negative. However, Aspect-Based Sentiment Classification (ABSC) demands a more nuanced approach, identifying the sentiment associated with specific "aspects" (e.g., "the food was great, but the service was slow").

Current approaches using Large Language Models (LLMs) typically rely on full fine-tuning or Parameter-Efficient Fine-Tuning (PEFT) like LoRA. These methods often update parameters across the entire architecture, which can be computationally expensive and introduce "noise" that leads to catastrophic forgetting. This research investigates whether specific "sentiment-specific" regions exist within LLMs and if targeting these regions through model editing can improve both efficiency and accuracy.

Methodology

The researchers propose a two-step surgical framework: Causal Tracing followed by Model Editing.

Overview of the Causal Tracing and Model Editing framework.

Causal Tracing Analysis

To locate where sentiment knowledge is stored, a "corrupt-and-restore" technique is employed. The model's input embeddings are intentionally corrupted with noise to break its prediction. Individual layers or tokens are then systematically restored to their clean state. A component is deemed "causally important" if its restoration significantly improves the model's ability to recover the correct sentiment prediction.

Model Editing

Once critical components are identified, the model is edited using two primary techniques:

  1. Weight-Based Editing: Low-rank updates are applied exclusively to the attention output projection matrices of the causally significant layers.
  2. Representation-Based Editing: The hidden states (residual stream) are manipulated at the specific positions of aspect terms to steer the model's output.

Experimental Setup

The framework was evaluated on Llama-2-7b across four diverse domains:

  • Restaurant & Laptop: Standard benchmarks from SemEval 2014/2015/2016.
  • Device: Digital device reviews.
  • Service: Web service reviews.

Key Insights: The Mid-Layer and Aspect Anchor

Causal tracing revealed a distinct pattern in how LLMs process sentiment for ABSC. Unlike many tasks where the final token is most influential, ABSC relies heavily on the hidden states of the aspect words themselves.

Causal tracing heatmap highlighting the importance of mid-layers and aspect tokens.

As shown in the heatmap, the "sweet spot" for sentiment information in Llama-2-7b resides in the mid-layers (10–15). Editing these layers at the aspect token position proved significantly more effective than editing the last token or random positions.

Results

The surgical editing approach demonstrated several advantages over traditional PEFT methods:

  • Accuracy: The method achieved an average in-domain accuracy of 87.6%, surpassing LoRA (86.5%) and LoReft (85.9%).
  • Efficiency: By focusing only on sentiment-specific layers, the method utilized only 0.006% of the model's parameters, compared to 0.826% for LoRA.
  • Generalization: The identified "sentiment-specific" neurons appeared consistent across domains, leading to more stable performance in out-of-domain testing.

Conclusion

This work demonstrates that LLMs possess localized, task-specific regions that can be targeted for highly efficient adaptation. By focusing on the mid-layers and aspect-token positions, model editing provides a "surgical" alternative to broad fine-tuning. Future work will explore whether these sentiment zones are consistent across different model architectures like Mistral or Falcon and how these techniques might be used to mitigate model bias.

Summary updated

BibTeX
@inproceedings{li2025exploring,
  author = {Shichen Li and Zhongqing Wang and Zheyu Zhao and Yue Zhang and Peifeng Li},
  title = {{Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34625},
}
Figure