Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments

Yu Gu, Xiang Deng, Yu Su

Abstract

Grounding large language models (LLMs) in real-world environments like databases and APIs typically relies on sequence-to-sequence generation of execution plans. However, this approach often suffers from syntactic errors and hallucinations. We present Pangu, a framework that decouples symbolic validity from neural plausibility. By using a symbolic agent to propose valid candidate plans and an LLM to discriminate among them, Pangu achieves state-of-the-art performance on Knowledge Base Question Answering (KBQA) tasks. Notably, smaller models like BERT-base under this framework can outperform much larger generative counterparts, while also enabling effective few-shot learning in complex environments.

Introduction

The challenge of "grounding"—linking natural language to structured environments such as databases, web APIs, or physical robots—is central to the utility of Large Language Models (LLMs). The prevailing paradigm treats grounding as a translation task: converting a user's request into a formal program (e.g., SQL or SPARQL). This "sequence-to-sequence" approach places a heavy burden on the model, requiring it to simultaneously master programming syntax, environment schemas, and logical reasoning without a global view of the environment. In large-scale settings like Freebase, these models frequently fail due to hallucinations or syntactic invalidity.

The Pangu Framework

Pangu (named after the primordial being who separated heaven and earth) addresses these issues by disentangling the symbolic and neural components of grounded understanding. Instead of generating code from scratch, the system uses a discriminative approach.

The Pangu framework separates neural plausibility from symbolic validity across different environments.

The framework consists of three primary components:

  1. The Symbolic Agent: This component ensures "validity." It interacts directly with the environment to incrementally construct plans. Because it follows the environment's strict rules, every proposed plan is guaranteed to be grammatical and executable.
  2. The Neural Language Model (LM): This component ensures "plausibility." Rather than writing code, the LM acts as a judge, scoring candidate plans based on how well they match the user's intent.
  3. The Search Process: Pangu employs a beam search. At each step, the symbolic agent proposes extensions to the current plan. The LM scores these candidates, and the top $K$ are retained for the next iteration.
An example of the Pangu search process (t=1 to t=4) for a KBQA task, showing how the LM scores candidate plans proposed by the symbolic agent.

Datasets and Evaluation

The researchers evaluated Pangu on three Knowledge Base Question Answering (KBQA) datasets:

  • GrailQA: A large-scale dataset testing three levels of generalization: i.i.d., compositional, and zero-shot.
  • GraphQ: A challenging, moderate-scale dataset with a non-i.i.d. setting.
  • WebQSP: A moderate-scale dataset focused on i.i.d. generalization for simpler questions.

Key Insights

  • Discrimination vs. Generation: Recognizing a correct answer is fundamentally easier than producing one. This allows even small models, such as BERT-base (110M parameters), to achieve record-breaking results when paired with a symbolic agent.
  • The Symbolic Shield: The symbolic agent acts as a filter, preventing the LLM from ever seeing or considering invalid plans. This removes the need for the model to learn complex negative constraints.
  • Global Evaluation: Unlike generative models that predict plans token-by-token (local normalization), Pangu evaluates entire sub-plans. This global view makes the system significantly more robust to early-stage errors.
  • Superior Generalization: Pangu is less prone to overfitting and maintains consistent performance across both seen and unseen program structures, whereas generative models often struggle with zero-shot generalization.

Results

Pangu with a T5-3B model established a new state-of-the-art across all three datasets. On the GraphQ dataset, Pangu with BERT-base improved the F1 score from 31.8% to 48.2%, surpassing much larger generative models. Furthermore, Pangu demonstrated the first successful few-shot in-context learning for KBQA; using Codex with only 10 examples, it outperformed all previous methods on GraphQ. The framework is also remarkably sample-efficient, requiring 10x to 100x less training data than standard generative baselines.

Conclusion and Future Work

While Pangu offers significant gains in accuracy and generalization, it comes with a trade-off in efficiency. The requirement to score multiple candidates at each step makes it roughly twice as slow as direct generation. Future research will explore training the discriminative model under weak supervision (using only final answers as rewards) and refining prompt engineering techniques, such as Chain-of-Thought, to further enhance its few-shot capabilities.

Summary updated

BibTeX
@inproceedings{gu2023don,
  author = {Yu Gu and Xiang Deng and Yu Su},
  title = {{Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.270},
}
Figure