Abstract
Wenyv Guan, Qianying Liu, Guangzhi Han, Bin Wang, Sujian Li. Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association.
2019.
Introduction
Complex generation tasks, such as translating natural language into executable code (Text2Code) or solving math word problems (MWP), often rely on a "coarse-to-fine" (coarse2fine) paradigm. This approach typically involves two stages: first, generating a "sketch"—a high-level template that abstracts away specific variables or numbers—and second, filling in the fine-grained details to produce the final output.
However, a significant limitation of existing coarse2fine models is their lack of a correction mechanism. If the model generates an erroneous sketch in the first stage, the second stage is forced to operate within that flawed framework, leading to inevitable failure. This "sketch trap" creates a performance ceiling, as the final output is strictly dependent on the accuracy of the initial rough draft.
Proposed Method: The Correction Mechanism
To address this, we propose an improved coarse2fine model that introduces a tunable control mechanism. The architecture consists of four primary components: a text encoder, a sketch decoder, a sketch encoder, and a final fine-stage decoder.
The core innovation is the introduction of a hyperparameter, $\lambda$ (lambda), which acts as a "control knob" for the fine-stage decoder. Instead of blindly following the sketch, the input to the final decoder ($i_t$) is a weighted combination of the sketch information and the model's previous predictions:
$$i_t = \lambda \cdot v_{t-1} + (1 - \lambda)emb(e_{t-1})$$
Where $v_{t-1}$ represents the state from the sketch encoder and $emb(e_{t-1})$ is the embedding of the previously generated token. By adjusting $\lambda$, the model can balance its reliance on the sketch against its internal predictions. A $\lambda$ of 1 forces the model to follow the sketch strictly, while lower values allow the model to "overrule" potential errors in the sketch based on the original input text.
Experimental Setup
The model was evaluated across three distinct tasks to demonstrate its versatility:
- Text2Logic (GEO): Generating logical queries from natural language.
- Text2Code (Django): Translating natural language descriptions into Python code.
- Math Word Problems (Math23k): Generating solvable algebraic equations from text.
Key Results and Insights
The improved model consistently outperformed baseline architectures across all datasets. Notably, this work represents the first successful application of the coarse-to-fine method to the Math Word Problem task.
- Text2Logic (GEO): Achieved 88.6% accuracy (+0.4% over baseline).
- Text2Code (Django): Achieved 76.1% accuracy (+2.0% over baseline).
- Math Word Problems: Achieved 67.9% accuracy (+1.2% over baseline).
A key finding is that the optimal value for $\lambda$ is task-dependent. Semantic parsing (Text2Logic) benefits from a high $\lambda$ (0.9), indicating high sketch reliability. Conversely, MWP tasks perform best with a much lower $\lambda$ (0.3), suggesting that math sketches are more prone to error or require greater flexibility during the fine-tuning stage.
Conclusion and Future Work
By allowing the model to "doubt" its intermediate steps, the proposed correction mechanism significantly improves generation performance. Future research will focus on automating the selection of the $\lambda$ hyperparameter, perhaps through reinforcement learning, and addressing the assumption of one-to-one alignment between sketches and final outputs in more complex, non-linear tasks.
Summary updated