TL;DR
The BIPro framework enables relatively small language models to outperform giants like GPT-4 in the complex art of traditional Chinese poetry. By mimicking the human process of iterative drafting and revision—using "block" generative models that can fill in the middle of a text—BIPro achieves state-of-the-art zero-shot generation quality without any domain-specific fine-tuning.
What Problem Does This Paper Address?
Generating traditional Chinese poetry is a "constrained writing" task of the highest difficulty. It requires adhering to the Pingshui rhyme scheme, which dictates strict rules for sentence length, tonal patterns (level vs. oblique tones), and rhyming.
Most modern Large Language Models (LLMs) like GPT-4 are autoregressive: they predict the next token based only on what came before. This creates a "dead-end" problem. If a model generates a beautiful third line that makes it impossible to satisfy the rhyming constraint in the fourth line, it has no way to go back and fix it. Humans, by contrast, don't just write straight through; we deliberate, produce drafts, and revise previous lines to better fit the overall structure. The authors argue that the lack of a "look-back" and revision mechanism is why even the most powerful general-purpose LLMs often fail to impress expert poets.
Methods
The authors introduce BIPro (Block Inverse Prompting), a framework designed to bring iterative refinement to AI generation. It relies on three main pillars:
- Block Generative Models: Unlike standard GPTs, models like GLM-10B use a "block" attention mechanism, allowing them to perform infilling. They can generate text given both a preceding context and a subsequent context.
- Inverse Prompting as a Scorer: Instead of just checking if a sentence is fluent, BIPro uses "Inverse Prompting." It calculates the probability that the generated poem would actually lead back to the original title/prompt ($p(prompt | generation)$). This serves as a proxy for how well the poem captures the intended theme.
- The Revise-and-Rewrite Loop:
- Revise: Immediately after a new sentence is generated, the model "looks back" at the previous sentence and attempts to modify it to better harmonize with the new one.
- Rewrite: Once a full draft is complete, the model systematically masks each sentence one by one and tries to generate a better version based on the rest of the poem. This cycle repeats until the quality (measured by the inverse prompt score) plateaus.
Datasets
- Base Model Training: The underlying model, GLM-10B-Chinese, was pre-trained on the Wudaocorpora, a large-scale general Chinese dataset. Crucially, it received no specific training on poetry.
- Evaluation Titles: The researchers used 42 open-domain titles suggested by reviewers.
- Human Benchmark: For the "Parallel Poem Generation" challenge, they used 87 human-authored poems from the "Daily Poem" section of the China Poetry website (zgshige.com) to serve as a gold standard for comparison.
Key Insights and Ideas
- Framework > Model Size: The most striking insight is that a 10-billion parameter model (GLM-10B) using BIPro outperformed the much larger GPT-4 and GLM-4. This suggests that for creative, highly structured tasks, the process of generation is just as important as the raw "intelligence" of the model.
- Mimicking Human Cognition: The "Revise" and "Rewrite" functions effectively simulate the "deliberate and planned creation" seen in human masterpieces. By allowing the model to retrospectively alter existing content, BIPro breaks the rigid linearity of standard LLMs.
- Direct vs. Inverse Evaluation: Standard perplexity (how "surprising" the text is to the model) is a poor measure of poetic quality. By using inverse prompting, the framework ensures the poem remains tightly coupled to its intended meaning, preventing the model from drifting into "hallucinated" but fluent nonsense.
Key Results
The researchers conducted double-blind human evaluations with proficient poets.
- Open-Domain Generation: BIPro achieved an overall score of 5.27/10, significantly beating GPT-4 (4.79) and specialized systems like Shisanbai (5.13) and Baidu Poetry Helper (4.76).
- Parallel Generation: When tasked with writing poems based on titles of existing human masterpieces, BIPro scored 5.54, narrowing the gap with high-quality human poetry (6.37) and again outperforming GPT-4 (4.98).
- Qualitative Excellence: In case studies (like the "Swallow" poem), reviewers noted that BIPro's output showed "deeper subtle human-like feelings" and used behavioral descriptions to imply a theme rather than just repeating keywords.
Limitations and Future Work
- Computational Cost: Iterative revision is expensive. BIPro requires roughly $O(mk)$ times more tokens than direct generation (where $m$ is the number of revision rounds and $k$ is the beam size). Generating one poem can take about a minute on an A100 GPU.
- Evaluation Bottleneck: Because poetry is subjective and artistic, the authors had to rely heavily on human experts. They acknowledge the difficulty of creating automated benchmarks that can truly capture "poetic soul."
- Potential for Abuse: The authors note that such high-quality generation could be misused for sarcasm, propaganda, or creating "cute slogans" for negative purposes, requiring future work on safety and supervision.
Summary updated