Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification

Shichen Li, Zhongqing Wang, Zheyu Zhao, Yue Zhang, Peifeng Li

Abstract

Aspect-Based Sentiment Classification (ABSC) requires a fine-grained understanding of specific entities within a text. Traditional fine-tuning of Large Language Models (LLMs) for this task is computationally expensive and often updates redundant parameters. This paper introduces a surgical model editing framework that identifies and modifies only the causally significant "sentiment-processing" neurons. Using causal tracing, we locate these critical regions in the mid-layers (10-15) at the aspect word positions. Our targeted editing approach achieves state-of-the-art performance on Llama-2-7b using only 0.006% of trainable parameters, demonstrating extreme efficiency without sacrificing accuracy.

Introduction

The challenge of Aspect-Based Sentiment Classification (ABSC) lies in its requirement for precise contextual understanding. For instance, in the sentence "The food was excellent, but the service was poor," a model must distinguish the positive sentiment for food from the negative sentiment for service.

Adapting massive LLMs for such tasks typically involves significant computational overhead. Even popular Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA can be inefficient, as they apply updates globally without regard for where specific task-relevant knowledge is stored. This research investigates whether we can identify the precise "brain regions" of an LLM responsible for sentiment and edit them directly.

Methodology: Causal Tracing and Targeted Editing

The proposed framework consists of two primary phases: identification and intervention.

1. Causal Tracing Analysis

To pinpoint where sentiment knowledge resides, the authors employ causal tracing. This involves three steps:

  • Clean Run: The model processes the input normally.
  • Corrupted Run: Noise is introduced to the input embeddings to degrade the model's performance.
  • Restoration Run: Hidden states from the clean run are systematically restored into the corrupted model at specific layers and token positions.

If restoring a specific state recovers the correct prediction, that state is identified as having a high "Average Indirect Effect" (AIE) and is deemed causally important for the task.

Overview of the Causal Tracing and Model Editing framework.

2. Model Editing

Once the critical layers and tokens are identified, the model is "edited" using two techniques:

  • Weight-based Editing: A low-rank adaptation is applied specifically to the attention output projection matrices within the identified layers.
  • Representation-based Editing: The residual stream is modified directly to steer the model's internal representations toward the correct sentiment polarity.

Locating Sentiment: The Mid-Layer Aspect Anchor

The causal tracing analysis yielded two significant insights regarding the internal mechanics of Llama-2-7b during ABSC tasks.

First, sentiment processing is not distributed uniformly. The analysis revealed a "sweet spot" in the mid-layers, specifically layers 10 through 15. Interventions in early or very late layers showed significantly less impact on the final prediction.

Second, the most influential hidden states are located at the token positions of the aspect words themselves, rather than the final token of the sequence. This suggests that the model anchors its sentiment reasoning on the specific aspect being evaluated.

Heatmap showing the Average Indirect Effect (AIE) across layers and tokens, highlighting the importance of the aspect token in mid-layers.

Experimental Results

The framework was evaluated across four domains: Restaurant, Laptop, Device, and Service reviews.

Performance and Efficiency

The targeted editing approach achieved an average in-domain accuracy of 87.6%, outperforming standard LoRA (86.5%) and even larger models like GPT-4o-mini (84.9%) in specific categories. Remarkably, this performance was achieved using only 0.006% of Llama-2-7b's parameters—a 130x reduction compared to typical PEFT configurations.

Generalization

In cross-domain evaluations (training on one domain and testing on another), the model maintained a robust average accuracy of 85.4%. This indicates that the identified "sentiment neurons" represent a generalized mechanism for processing polarity within the LLM that is consistent across different topics.

Conclusion and Future Work

This research demonstrates that LLMs can be adapted for complex NLU tasks with surgical precision. By identifying and editing only the causally relevant parameters, researchers can achieve superior performance with a fraction of the computational cost. Future work includes extending this methodology to other tasks like Named Entity Recognition (NER) and investigating how these "knowledge maps" vary across different LLM architectures.

Summary updated

BibTeX
@inproceedings{li2025exploring,
  author = {Shichen Li and Zhongqing Wang and Zheyu Zhao and Yue Zhang and Peifeng Li},
  title = {{Exploring Model Editing for LLM-based Aspect-Based Sentiment Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34625},
}
Figure