Self-Correction Makes LLMs Better Parsers

Ziyan Zhang, Yang Hou, Chen Gong, Zhenghua LiEMNLP 2025

TL;DR

Large Language Models (LLMs) struggle with syntactic parsing, often producing "flatter" and less accurate parse trees than traditional parsers. This paper introduces a novel self-correction method that leverages existing treebanks to guide LLMs in identifying and correcting parsing errors, significantly improving their performance without additional training.

What Problem Does This Paper Address?

Syntactic parsing, the process of analyzing the grammatical structure of sentences, is a fundamental task in Natural Language Processing (NLP). While LLMs have shown impressive capabilities across many NLP tasks, they exhibit a surprising weakness in syntactic parsing. Studies suggest LLMs produce parse trees that are less deep and structurally less accurate than those generated by specialized, non-LLM parsers. This paper aims to understand why LLMs falter in parsing and proposes a method to enhance their parsing abilities by enabling them to learn from existing linguistic knowledge without requiring further training.

Methods

The core of this paper is a self-correction method designed to improve LLM parsing. The approach has two main stages:

  1. Sentence Mismatch Correction: This initial step addresses errors where the LLM's generated parse tree includes words not present in the original sentence or omits words from it. The method identifies these discrepancies and provides specific hints to the LLM to generate a corrected output with accurate word alignment.
  2. Tree Structure Correction: After ensuring word alignment, this stage focuses on correcting structural errors within the parse trees. It involves:
  3. Identifying Errors Based on Rules: The method extracts grammar rules from the LLM's predicted parse tree and checks if these rules exist in a reference treebank. Rules not found are flagged as potential errors.
  4. Processing Rules: For identified errors, the method processes rules to find similar, valid rules in the treebank. This processing is tailored to different error types (span, label, flatness, deepness).
  5. Ranking Rules: The similarity between processed rules and gold-standard rules is measured using the Longest Common Subsequence (LCS) of constituent labels. Top-ranked, similar rules are selected.
  6. Guiding LLM: Examples of these high-ranked rules are sampled from the treebank and used as hints in a prompt to guide the LLM in correcting its previous parse tree.

This method is novel in its direct use of existing treebanks as external knowledge sources for error identification and correction, and its focus on dynamically searching for relevant rules based on error characteristics.

Datasets

The authors evaluated their method on the following datasets:

  • Penn Treebank (PTB): A standard English constituency parsing dataset.
  • Chinese Penn Treebank 5 (CTB5): A Chinese constituency parsing dataset.
  • Multi-domain Constituent Treebank (MCTB): A dataset covering various domains (Dialogue, Forum, Law, Literature, Review) used for cross-domain evaluation.

For few-shot learning, five examples were sampled from the training sets of PTB and CTB5. The reference treebanks used for self-correction were the training sets of PTB and CTB5.

Key Insights and Ideas

  • LLMs are "shallow parsers": The analysis reveals that LLMs struggle with deep structural understanding, often producing flatter parse trees than expected.
  • Lack of "known rules" is a key limitation: The performance gap between LLMs and traditional parsers is attributed to LLMs' limited ability to leverage grammar rules present in treebanks, especially during few-shot learning.
  • Self-correction is effective without retraining: The proposed method demonstrates that LLMs can significantly improve their parsing capabilities by learning from external linguistic knowledge (treebanks) through a guided self-correction process, without requiring costly fine-tuning.
  • Span errors are prevalent but indirectly addressed: While span errors (incorrect word boundaries) are the most frequent, the top-down correction approach indirectly helps reduce them by correcting higher-level structures.
  • LLMs can learn beyond explicit training: The method not only improves the use of known rules but also enhances the accuracy of predicting unknown rules, suggesting LLMs can generalize structural patterns.

Key Results

  • The self-correction method achieved substantial and consistent improvements across all datasets and LLMs tested.
  • On the PTB and CTB5 datasets (in-domain), the method improved F1 scores by over 10 points compared to direct LLM parsing.
  • For cross-domain parsing (MCTB), significant improvements were also observed, demonstrating robustness.
  • Compared to a previous method that uses chunking, this self-correction approach with GPT-4 achieved a higher F1 score on PTB.
  • The method significantly reduced error types like NP Internal Structure and PP Attachment.
  • The correction process has a low wrong correction rate (around 4%), with correct corrections far outweighing errors.

Limitations and Future Work

  • Determining similarity: The paper acknowledges the need to explore whether rules similar but not identical to gold trees can effectively guide LLM corrections.
  • Learning assessment: Further investigation is needed to assess the extent to which LLMs truly "learn" from the provided examples.
  • Span error direct correction: While indirectly addressed, directly correcting span errors remains a challenge.
  • Specific error types: Certain error types, like VP Attachment and NP Attachment, showed limited improvement, suggesting these remain challenging for LLMs.

Summary updated

BibTeX
@inproceedings{zhang2025self,
  author = {Ziyan Zhang and Yang Hou and Chen Gong and Zhenghua Li},
  title = {{Self-Correction Makes LLMs Better Parsers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure