Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments

Yu Gu, Xiang Deng, Yu Su

In the pursuit of making Large Language Models (LLMs) more functional, "grounding"—the process of connecting a model's linguistic capabilities to real-world environments like databases, web pages, or physical robots—remains a significant challenge. Traditionally, researchers have approached grounding as a generation problem, requiring models to write complex SQL queries or robot control plans from scratch. However, researchers at The Ohio State University propose a paradigm shift: Pangu, a framework that moves the role of Language Models from generating plans to discriminating between them.

The Problem with Generative Grounding

Current approaches to grounded language understanding typically rely on a "Sequence-to-Sequence" (Seq2Seq) methodology. A user provides a natural language request (e.g., "Find all species of a certain butterfly genus"), and the model attempts to generate a formal program, such as SPARQL or SQL, to fulfill it.

This approach places a heavy cognitive burden on the LM, requiring it to simultaneously manage user intent, strict programming syntax (grammaticality), and the specific constraints of a dynamic environment (faithfulness). When LLMs fail in these tasks, they often "hallucinate" invalid code or reference non-existent entities. Even constrained decoding techniques, which force models to follow a grammar, often suffer from "shortsightedness," where local token-level choices lead to global logical failures.

The Pangu Framework

Pangu, named after the primordial being in Chinese mythology who separated heaven and earth, addresses these issues by disentangling the symbolic (environment rules) from the neural (linguistic intuition).

Overview of the Pangu framework separating neural scoring from symbolic plan generation across various environments.

As shown in the architecture above, Pangu utilizes two distinct components:

  1. The Symbolic Agent: Instead of the LM guessing, a symbolic agent explores the environment (such as a Knowledge Base) to incrementally construct valid sub-plans. Because this agent interacts directly with the environment, every proposed plan is guaranteed to be syntactically correct and executable.
  2. The Neural Discriminator: The LM acts as a judge. Given a user utterance and a candidate sub-plan, it assigns a plausibility score. This allows the LM to focus on its core strength—semantic matching—while the symbolic agent handles the rigid task of rule-following.

Search and Discrimination

Pangu operates through a beam search process. At each step, the symbolic agent proposes extensions to the current plan based on the environment's schema. The LM then scores these candidates, and only the top-$K$ most plausible plans are retained for the subsequent step.

Step-by-step beam search process where the symbolic agent proposes plans and the neural model scores them.

This discriminative approach offers several advantages:

  • Global Perspective: Unlike generative models that pick one token at a time, Pangu evaluates entire sub-plans, preventing the catastrophic local errors common in Seq2Seq models.
  • Sample Efficiency: It is computationally and cognitively simpler for a model to recognize a correct answer than to produce one from scratch.
  • Uniformity: Pangu provides a "plug-and-play" interface, allowing researchers to use encoder-only (BERT), encoder-decoder (T5), or decoder-only (GPT) models as the scorer without altering the framework.

Experimental Evaluation and Results

The researchers primarily evaluated Pangu on Knowledge Base Question Answering (KBQA) using massive environments like Freebase. Testing was conducted across several datasets, including GRAILQA (testing compositional and zero-shot generalization), GRAPHQ (testing novel structures with small training sets), and WEBQSP (focused on simpler questions).

Key findings include:

  • State-of-the-Art Performance: Pangu established new records across all three KBQA datasets.
  • Model Efficiency: A BERT-base model (110M parameters) using Pangu outperformed generative models with over 3 billion parameters on the GRAPHQ dataset.
  • Few-Shot Breakthrough: Pangu enabled the first effective few-shot in-context learning for KBQA. Using Codex with just 10 examples, Pangu outperformed previous SOTA models trained on the full GRAPHQ dataset.
  • Generalization: On GRAILQA, Pangu showed a significant lead in zero-shot domains, proving that discrimination generalizes better to unseen environments than direct generation.

Limitations and Future Work

While Pangu represents a significant step forward, it is currently slower than direct generation due to the need for multiple scoring passes at every step (roughly 2x slower than fast baselines). Future work will explore "weak supervision" to learn from final answers rather than perfect plans, and investigate advanced prompting techniques like "Chain-of-Thought" to further enhance performance. Additionally, while Pangu naturally improves controllability by excluding invalid plans, its potential for broader safety and security applications remains an area for future study.

Summary updated

BibTeX
@inproceedings{gu2023don,
  author = {Yu Gu and Xiang Deng and Yu Su},
  title = {{Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.270},
}
Figure