Introduction
Aspect Sentiment Quad Prediction (ASQP) is a complex information extraction task that requires identifying four interconnected elements: the aspect term, the opinion term, the aspect category, and the sentiment polarity. While Large Language Models (LLMs) like GPT-4 have shown remarkable semantic understanding, they often struggle with the precise boundary detection required for extracting specific terms. Current prompting strategies typically rely on semantic similarity to retrieve demonstrations, but this ignores the structural patterns that guide term extraction. This paper introduces SimRP, a retrieval prompting strategy that leverages both syntactic and semantic similarity to improve ASQP performance.
The SimRP Framework
The core innovation of SimRP is the integration of syntactic structure into the example retrieval process. The framework operates through a novel vectorization method called Syn2Vec and a two-stage retrieval pipeline.
Syn2Vec: Capturing Sentence Structure
To make grammatical structures searchable, the authors propose Syn2Vec. This method converts a sentence's constituent tree—a hierarchical map of its grammatical components—into a sparse vector. By applying Inverse Document Frequency (IDF) weighting, the model prioritizes rare and highly informative grammatical patterns. This ensures that the retrieved examples share a similar structural "skeleton" with the target sentence.
The Retrieval Pipeline
SimRP employs a dual-stage selection process:
- Syntactic Filtering: Syn2Vec identifies the top-$k$ most syntactically similar candidates from the training set.
- Semantic Refinement: From this structural subset, a semantic model (such as Sentence-BERT) selects the most contextually relevant examples.
These "dual-similar" examples are then used as few-shot demonstrations, providing the LLM with both the right context and the right structural template to follow.
Evaluation via CsT-based Fuzzy Metric
A significant challenge in ASQP is the "metric gap." Traditional evaluation relies on exact string matching, which penalizes LLMs for including minor descriptive words (e.g., "a fair tab" vs. "fair"). To address this, the authors introduced the Constituent Subtree (CsT)-based fuzzy metric. This metric considers a prediction correct if the extracted term falls within the same grammatical subtree as the ground truth, offering a more nuanced and human-aligned assessment of model utility.
Experimental Results
The SimRP framework was evaluated on the Rest15 and Rest16 benchmarks. The results demonstrate that SimRP consistently outperforms existing state-of-the-art (SOTA) generative models.
As shown in the results, SimRP achieved an average F1 score improvement of 2.20% over previous SOTA models like MvP and GenDA. Furthermore, when applying the CsT-based fuzzy metric, the performance of SimRP jumps significantly (e.g., from 63.42 to 68.42 on Rest16). This suggests that LLMs are often more capable than rigid metrics suggest, and structural prompting helps bridge the gap between raw capability and precise extraction.
Conclusion
SimRP demonstrates that syntax is a critical "missing link" in prompt engineering for extraction tasks. By providing models with structurally similar demonstrations, SimRP enables more accurate identification of aspect and opinion terms. Coupled with more rational evaluation metrics, this approach reveals the true potential of LLMs in complex sentiment analysis tasks. Future work will explore the application of structural prompting to other domains, such as named entity recognition and relation extraction.
Summary updated