Syntactic parsers trained on edited news lose much of their accuracy on dialogue, forums, and reviews. The accuracy they lose is concentrated in a few decisions, above all where a prepositional phrase or a noun modifier attaches. That concentration suggests a cheap fix. Let a strong parser produce a tree in one pass, then send only the doubtful parts of that tree through a second model that re-decides them in light of the rest of the tree. Keep repeating until nothing changes. Well-formed sentences would cost one pass; badly shifted sentences would get more work, and only in the regions that need it.
The catch is that a refiner, meaning a model that takes a draft tree and outputs a corrected one, learns to fix the mistakes it saw in training. If those are news-domain mistakes, it may do little on web text, or even pull correct but unusual attachments back toward news-style structure. This note proposes to train the refiner on mistakes the base parser makes on synthetic target-domain sentences whose correct trees are known by construction. The refiner then re-decides flagged regions only, with a learned signal that marks which parts of the tree to revisit and when to stop. The claim to test is that refinement pays off across domains only when its training errors look like the target's errors. Given that, refinement concentrated on flagged regions should keep most of the gain of refining the whole tree at a fraction of the cost.
What the literature establishes, and what it leaves open
Three results motivate the proposal. First, domain shift can hurt parsers more than a change of language: a newswire parser falls from 90.6% to 73.6% on Twitter, a larger drop than English-to-Portuguese transfer (What I think when I think about treebanks). Second, the damage comes mainly from structure rather than vocabulary. Challenges to Open-Domain Constituency Parsing shows that differences in grammar-rule and grandparent-chain distributions predict the accuracy drop much better than word-distribution differences. Even with pretrained contextual embeddings, parsers fall from about 95 F1 on news to about 85 on dialogue and 84 on reviews. Third, the errors are local. Parser Evaluation Using Derivation Trees finds head accuracy for nominal modifiers falling from 91.6% on WSJ to 42.1% on English Web Treebank Answers, and PP attachment falling from 90.5% to 80.7%. Simple noun phrases stay near 99%. Together these say where compute should go, but no parser exploits that.
Two lines of work supply the method. Recursive Non-Autoregressive Graph-to-Graph Transformer (RNGTr) feeds the previous pass's dependency arcs into the attention of the next pass. It refines for a fixed number of passes (typically 3) over the whole sentence and finishes with a maximum-spanning-tree decode that guarantees a valid tree, reaching 95.01 labeled attachment on PTB. It is evaluated only on in-domain splits, refines every token uniformly, and leaves stopping criteria to future work. Reflection-Window Decoding shows, for ordinary left-to-right text generation, that refinement triggered by uncertainty can touch only 3.5–5.5% of tokens and still beat full beam search. It has no notion of trees, and its trigger is next-token entropy. We take RNGTr's idea of conditioning on the current tree and Reflection-Window's idea of refining selectively. We move both to constituency trees under domain shift, and we replace the entropy trigger, which there is reason to distrust out of domain.
Two lines of work supply the data and the competition. Contrastive Learning on LLM Back Generation Treebank (CTPT) builds a synthetic target-domain treebank. It freezes a bare tree skeleton from PTB and asks an LLM to write domain-flavored text that fits it, so every sentence comes with a valid gold tree. Training on 10k such sentences with span-level contrastive learning gives 88.52 average F1 on the five-domain MCTB benchmark. Cross-domain Constituency Parsing by Leveraging Heterogeneous Data (CLDNT) reaches 87.78 through multi-task prefix tuning, against 87.14 for a plain chart parser. Both methods decode once. We reuse CTPT's treebank for a different purpose: not to train a better one-pass parser, but to harvest the base parser's own mistakes on target-like text as supervision for the refiner.
The file also contains results that rule out some designs. LLMs prompted to parse directly score 29.33 (GPT-4) and 22.99 (ChatGPT) F1 on MCTB (CTPT). Reflection-Window's authors note that refinement cannot fix what the model does not know. So the refiner must be a small structured model that always outputs valid trees, not an LLM critiquing its own output. The latency literature (Meta-Reflection, which compresses reflection into one pass and cuts first-token latency from 951 to 246 ms) reminds us that any multi-pass gain has to be weighed against a one-pass model given the same compute.
Core Claim
Our first claim is about training data. A tree refiner trained on the base parser's errors over back-generated target-domain sentences will improve MCTB F1 on top of the strongest one-pass parser. The same refiner trained on source-domain errors will give little or no out-of-domain gain.
Our second claim is about compute. When revisits are limited to regions a learned per-span flag marks as likely wrong, and passes halt once no span is flagged, the refiner should keep at least 90% of the full-tree refiner's gain while re-deciding a small minority of spans. Its average number of passes should rise with a domain's structural divergence from news: few on Law, most on Dialogue and Review.
Research Program
The base parser is a span-based chart parser trained on PTB. We use the CTPT parser itself, so that gains are measured on top of the best available one-pass system.
The refiner reuses the same pretrained encoder, plus a few extra layers. Each span's attention is biased by an embedding of the current tree: whether the span is a bracket, its label, and its parent's label. This is the constituency analogue of how RNGTr injects arcs into attention. The refiner outputs new span scores and a per-span probability that the current decision is wrong.
At inference, spans flagged above a threshold, together with their immediate parents and siblings, are opened for revision. All other brackets are held fixed as constraints. A CKY search, the standard dynamic-programming algorithm for finding the best-scoring tree, is rerun only over the opened spans. Every intermediate output is therefore a valid tree, with no invalid brackets of the kind LLM parsers produce. The loop stops when nothing is flagged or when a cap of four passes is reached.
Training data comes from running the base parser on the CTPT synthetic treebank, which has 10k sentences across the five MCTB domain styles with gold trees. We also add drafts with injected attachment errors of the types Kulick et al. identify, so the refiner sees multi-step trajectories rather than single corrections. The flag head is trained against whether each draft span matches gold. Before any refinement result is interpreted, we check the data itself. We compare the base parser's error-type mix on the synthetic sentences (PP, nominal modifier, coordination, clause boundary) with its mix on a small labeled MCTB slice. CTPT inherits PTB skeletons, so synthetic errors may be too news-like, and this is the proposal's main risk. If the two mixes diverge sharply, we treat the synthetic data as a lower bound and report that.
For a dependency replication, the same selective loop runs inside RNGTr on UD English EWT, with head-rule conversions, and on EWT Answers. There we can test directly whether revisits land on nominal-modifier and PP arcs.
Baselines
The naive reference is the plain one-pass chart parser. CLDNT reports 87.14 average MCTB F1 for it, and we rerun it with our encoder for exact comparability.
The strongest existing method is CTPT at 88.52 average F1 (CTPT). It sees the same 10k synthetic sentences we use. To keep the comparison fair, CTPT also gets the refiner's extra parameters and the extra training passes: we train a CTPT parser with the added layers at a matched number of gradient steps. Beating it means a gain on average MCTB F1 that holds across three seeds and is significant under paired bootstrap. The gain should also be concentrated in Dialogue, Forum, and Review, where structural divergence is largest.
The closest iterative competitor is a full-tree refiner in the style of RNGTr, run for a fixed 3 passes and trained on the same synthetic errors. It defines the accuracy ceiling our selective version must stay close to, and the compute cost it must undercut. We report spans re-decided, passes, and wall-clock time at batch size 1 and at throughput batch sizes.
Two ablations isolate the key ingredients. The first swaps the refiner's training errors: PTB-only errors versus synthetic target-domain errors, with architecture and budget held fixed. This tests the first claim. The second swaps the trigger. The learned flag is compared with the base parser's span-marginal entropy and with random flagging at matched revisit rates. This tests whether the flag carries information that the parser's own confidence, possibly miscalibrated out of domain, does not.
Thresholds are tuned on a held-out 20% of each MCTB domain, and results are reported on the remainder. Every baseline gets the same tuning budget.
Why not just…?
Why not just run RNGTr-style refinement and stop when no decision changes? This is the strongest threat to the latency claim. It gives dynamic stopping for free and may be nearly as cheap as learned flags. We run it as a baseline. If it matches our selective loop on both accuracy and spans re-decided, the compute contribution collapses and the proposal reduces to the first claim about training data. The flag earns its place only if it avoids revisiting spans that are stable but still recomputed under the "no change" rule, which we measure directly.
Why not just put the synthetic data into a better one-pass parser? CTPT already does exactly this, and a larger CTPT with matched training might absorb whatever the refiner learns. The matched-capacity CTPT baseline settles this. If the gap closes, the lesson is that conditioning on a draft tree adds nothing beyond extra capacity. That would be a useful negative result to report.
Why not just rerank a k-best list of trees? In text generation, Self-Refinement Tuning found that editing beats best-of-N, and Reflection-Window found that selective refinement beats beam search (66.67% versus 46.3% win rate on MT-Bench). Neither result was obtained on trees. We therefore run k-best CKY with a reranker trained on the same synthetic data, at matched wall-clock time. The expectation is that the correct attachment for a non-canonical PP often never appears in a short k-best list, while local re-decision can reach it.
Why not just have an LLM critique and repair the tree? The GPT-4 result of 29.33 F1 on direct parsing (CTPT), together with the knowledge-ceiling caveat from Reflection-Window, makes this unpromising. It also adds the latency this work tries to avoid. We include one run as a sanity check: GPT-4 repairs CTPT output, with changes that would make the tree invalid rejected.
Decisive Checks
Four results decide the proposal. First, the training-error ablation: a refiner trained on PTB errors should add near zero F1 on MCTB, while one trained on synthetic errors should add a measurable gain over matched-capacity CTPT. Second, selectivity: the selective loop should keep at least 90% of the full refiner's gain while re-deciding a minority of spans, and should beat the "stop when unchanged" rule on spans re-decided. Third, allocation: per-domain average passes should correlate with the structural divergence scores of Challenges to Open-Domain Constituency Parsing. On EWT Answers, most revisits should fall on nominal-modifier and PP decisions. Fourth, the data audit: we compare the error-type mix on synthetic sentences with the mix on real MCTB text, to show whether the first check's outcome reflects the refiner itself or the limits of skeleton-based synthetic data.
References
- What I think when I think about treebanks
- Challenges to Open-Domain Constituency Parsing
- Parser Evaluation Using Derivation Trees: A Complement to evalb
- Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement
- Reflection-Window Decoding: Text Generation with Selective Refinement
- Contrastive Learning on LLM Back Generation Treebank for Cross-domain Constituency Parsing
- Cross-domain Constituency Parsing by Leveraging Heterogeneous Data
- Meta-Reflection: A Feedback-Free Reflection Learning Framework
- Teaching Language Models to Self-Improve by Learning from Language Feedback