Graph-Structures Matching for Review Relevance Identification

Lakshmi Ramachandran, Edward GehringerEMNLP 2013

TL;DR

This paper introduces a novel graph-based approach to automatically identify whether a review is relevant to the submission it's evaluating. By representing texts as word order graphs and leveraging semantic similarity metrics, the method achieves a 66% accuracy in relevance identification, outperforming traditional text overlap methods. This work is crucial for improving the quality and utility of peer reviews in academic and educational settings.

What Problem Does This Paper Address?

In many fields, including academia, education, and grant funding, peer reviews are critical for decision-making. The usefulness of a review, however, hinges on its relevance to the work being reviewed. A review that doesn't accurately address the submission's content can be unhelpful, leading to misunderstandings or missed opportunities for improvement. Identifying this relevance automatically is challenging because reviewers often paraphrase or rephrase ideas, using different vocabulary and sentence structures than the original submission. Traditional text matching methods that rely on exact word matches often fail to capture these nuanced relationships, leading to inaccurate relevance assessments. This paper aims to bridge this gap by developing a more sophisticated method for identifying review relevance that accounts for semantic and syntactic variations.

Methods

The core of this paper's contribution lies in its innovative use of word order graphs and lexico-semantic matching.

  • Word Order Graphs: Instead of treating text as a bag of words, the authors represent it as a graph where vertices represent words or phrases, and edges capture their sequential and grammatical relationships. This structure is crucial for identifying changes in word order and sentence structure, which are common in paraphrasing. They detail a process for generating these graphs, involving text segmentation, part-of-speech tagging, and a heuristic phrase chunking technique. Edges are further enriched with dependency labels (e.g., SBJ for subject-verb) to capture grammatical roles.
  • Lexico-Semantic Matching: To determine the similarity between two texts (a submission and a review), they compare their respective word order graphs using a multi-faceted matching approach:
  • Phrase or Token Matching: Compares individual vertices (words or phrases) based on semantic relatedness, going beyond exact matches to include synonyms, hypernyms, hyponyms, and meronyms, leveraging WordNet for this.
  • Context Matching: Compares edges (representing relationships between vertices) to capture more contextual information. This includes:
  • Ordered Match: Compares edges with the same vertex types and order.
  • Lexical Change Match: Compares edges where the order of vertices is flipped, allowing for the detection of paraphrases involving word order changes.
  • Nominalization Match: Identifies noun forms derived from verbs or adjectives.
  • Sentence Structure Matching: Compares "double edges" (representing complete segments like subject-verb-object) to capture more complex structural similarities and voice changes.

The overall relevance score is an average of the scores obtained from these three matching types, weighted by their contribution.

Datasets

The authors used review-submission pairs from Expertiza, a web-based collaborative learning application. They selected 986 pairs, including both relevant and non-relevant cases, for their study. A portion of this data was manually annotated by two human annotators to establish ground truth and measure inter-annotator agreement.

Key Insights and Ideas

  • Word order matters: Simply matching words or even semantic concepts isn't enough. The order in which words and phrases appear, and the grammatical relationships they form, are vital for accurately identifying paraphrased content. Word order graphs capture this crucial dimension.
  • Beyond exact matches: The reliance on WordNet for semantic relatedness (synonyms, hypernyms, etc.) is key to uncovering relevance when reviewers don't use the exact same terminology as the submission.
  • Contextual matching improves accuracy: Individual matching types (phrase, context, sentence structure) contribute to the overall relevance score, but their combined effect, especially when incorporating context (ordering and grammatical roles), significantly boosts performance compared to any single method.
  • Dependency trees fall short for this task: While dependency trees capture grammatical relationships, they don't explicitly encode word order. This limitation makes them less effective than word order graphs for identifying paraphrases that involve shuffling of words or phrases.

Key Results

  • The proposed graph-based approach achieved an accuracy of 66% and an f-measure of 0.67 in identifying review relevance.
  • The approach demonstrated a high recall of 0.71, indicating a good agreement with human relevance judgments.
  • The combined approach (phrase + context + sentence structure matching) outperformed individual matching types.
  • Compared to a simple text overlap approach (1-4 gram overlap), the graph-based method achieved a higher f-measure (0.67 vs. 0.59) and a lower false negative rate.
  • The word order graph representation was also faster than dependency tree representations for matching.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that simple phrase matching, without context, can sometimes inflate the degree of relatedness, potentially misclassifying reviews.
  • The performance of context matching was sometimes negatively impacted by lexical and nominalization matches, as not all reviews exhibit these specific types of changes.
  • The study focused on single and double edges, not more contiguous edges, for text matching.
  • Future Work:
    • The authors plan to improve their review assessment system by providing more detailed textual feedback to reviewers, pointing to specific instances of reviews that need improvement, in addition to numeric scores.
  • They aim to further enhance the format of their output to make it easier for reviewers to interpret and act upon the feedback.

Summary updated

BibTeX
@inproceedings{ramachandran2013graph,
  author = {Lakshmi Ramachandran and Edward Gehringer},
  title = {{Graph-Structures Matching for Review Relevance Identification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
}
Figure