SimRP: Syntactic and Semantic Similarity Retrieval Prompting Enhances Aspect Sentiment Quad Prediction

Zhongquan Jian, Yanhao Chen, Jiajian Li, Shaopan Wang, Xiangjian Zeng, Junfeng Yao, Xinying An, Qingqiang WuAAAI 2025 · DOI

Introduction

Aspect Sentiment Quad Prediction (ASQP) is a complex information extraction task that requires identifying four interconnected elements: the aspect term, the opinion term, the aspect category, and the sentiment polarity. While Large Language Models (LLMs) like GPT-4 have shown remarkable semantic understanding, they often struggle with the precise boundary detection required for extracting specific terms. Current prompting strategies typically rely on semantic similarity to retrieve demonstrations, but this ignores the structural patterns that guide term extraction. This paper introduces SimRP, a retrieval prompting strategy that leverages both syntactic and semantic similarity to improve ASQP performance.

The SimRP Framework

The core innovation of SimRP is the integration of syntactic structure into the example retrieval process. The framework operates through a novel vectorization method called Syn2Vec and a two-stage retrieval pipeline.

Syn2Vec: Capturing Sentence Structure

To make grammatical structures searchable, the authors propose Syn2Vec. This method converts a sentence's constituent tree—a hierarchical map of its grammatical components—into a sparse vector. By applying Inverse Document Frequency (IDF) weighting, the model prioritizes rare and highly informative grammatical patterns. This ensures that the retrieved examples share a similar structural "skeleton" with the target sentence.

The SimRP pipeline uses Syn2Vec to identify sentences with similar constituent trees, facilitating better pattern matching for LLMs.

The Retrieval Pipeline

SimRP employs a dual-stage selection process:

  1. Syntactic Filtering: Syn2Vec identifies the top-$k$ most syntactically similar candidates from the training set.
  2. Semantic Refinement: From this structural subset, a semantic model (such as Sentence-BERT) selects the most contextually relevant examples.

These "dual-similar" examples are then used as few-shot demonstrations, providing the LLM with both the right context and the right structural template to follow.

Evaluation via CsT-based Fuzzy Metric

A significant challenge in ASQP is the "metric gap." Traditional evaluation relies on exact string matching, which penalizes LLMs for including minor descriptive words (e.g., "a fair tab" vs. "fair"). To address this, the authors introduced the Constituent Subtree (CsT)-based fuzzy metric. This metric considers a prediction correct if the extracted term falls within the same grammatical subtree as the ground truth, offering a more nuanced and human-aligned assessment of model utility.

Experimental Results

The SimRP framework was evaluated on the Rest15 and Rest16 benchmarks. The results demonstrate that SimRP consistently outperforms existing state-of-the-art (SOTA) generative models.

Performance comparison of SimRP against baseline methods on Rest15 and Rest16 datasets, including the impact of the CsT-based fuzzy metric.

As shown in the results, SimRP achieved an average F1 score improvement of 2.20% over previous SOTA models like MvP and GenDA. Furthermore, when applying the CsT-based fuzzy metric, the performance of SimRP jumps significantly (e.g., from 63.42 to 68.42 on Rest16). This suggests that LLMs are often more capable than rigid metrics suggest, and structural prompting helps bridge the gap between raw capability and precise extraction.

Conclusion

SimRP demonstrates that syntax is a critical "missing link" in prompt engineering for extraction tasks. By providing models with structurally similar demonstrations, SimRP enables more accurate identification of aspect and opinion terms. Coupled with more rational evaluation metrics, this approach reveals the true potential of LLMs in complex sentiment analysis tasks. Future work will explore the application of structural prompting to other domains, such as named entity recognition and relation extraction.

Summary updated

BibTeX
@inproceedings{jian2025simrp,
  author = {Zhongquan Jian and Yanhao Chen and Jiajian Li and Shaopan Wang and Xiangjian Zeng and Junfeng Yao and Xinying An and Qingqiang Wu},
  title = {{SimRP: Syntactic and Semantic Similarity Retrieval Prompting Enhances Aspect Sentiment Quad Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34601},
}
Figure