Semantic-aware Contrastive Learning for More Accurate Semantic Parsing

Shan Wu, Chunlei Xin, Bo Chen, Xianpei Han, Le Sun

Introduction

Semantic parsing—the process of translating natural language into formal Meaning Representations (MRs) like SQL or lambda calculus—is a cornerstone of modern natural language interfaces. While standard models have made significant strides, they often struggle with fine-grained logical distinctions. Researchers have introduced SemCL (Semantic-aware Contrastive Learning), a framework designed to move beyond token-by-token training by treating MRs as holistic logical units.

The Semantic Blind Spot of MLE

Traditionally, semantic parsers are trained using Maximum Likelihood Estimation (MLE), which optimizes the model to predict the next token in a sequence. However, MLE suffers from a "semantic blind spot": it treats all token errors with equal weight based on string distance, ignoring the logical impact.

In formal logic, a single character change (e.g., swapping > for <) can completely invert a statement's meaning. Analysis of the OVERNIGHT dataset reveals that over 74% of standard parser errors are within an edit distance of only 3 from the ground truth. These models are "close" in sequence space but "miles away" in logical space due to a lack of sequence-level semantic awareness.

Examples of minor string changes leading to major logical differences.

The SemCL Framework

To bridge this gap, SemCL introduces a contrastive learning objective that forces the model to distinguish between logically distinct outputs, even when they appear similar. The framework consists of three core components:

1. Multi-level Online Instance Sampling

Instead of training only on "gold" pairs, SemCL samples candidates during training and partitions them into three ranks:

  • Rank 0 (Positive): Ground-truth annotations and known logical aliases.
  • Rank 1 (Vague): Paraphrases or MRs that yield the correct execution result despite structural differences.
  • Rank 2 (Negative): MRs that result in incorrect execution or logical contradictions.

2. Semantic-aware Similarity Functions

The model employs specialized functions to measure the alignment between an utterance $x$ and an MR $y$. The most sophisticated version, MR-Conditioned Similarity, utilizes the decoder's attention context to pinpoint which parts of the utterance contributed to specific segments of the generated MR.

3. Ranked Contrastive Loss

A specialized loss function pulls Rank 0 instances closer to the utterance representation while pushing Rank 2 instances away. Crucially, it manages Rank 1 (vague) instances to prevent them from introducing noise during the learning process.

Overview of the Semantic-aware Contrastive Learning (SemCL) architecture.

Experimental Evaluation

The authors evaluated SemCL on two prominent benchmarks:

  • OVERNIGHT: A multi-domain dataset requiring precise logical mapping across eight distinct areas (e.g., Basketball, Recipes).
  • GEOGRANNO: A version of the GEO dataset labeled with lambda DCS, testing generalization with limited training data.

Key Results

SemCL achieved state-of-the-art (SOTA) performance on both datasets:

  • OVERNIGHT: Accuracy improved to 81.1%, a significant increase from the 75.7% vanilla SEQ2SEQ baseline.
  • GEOGRANNO: Reached 73.4% accuracy, surpassing previous benchmarks.

Notably, these gains were achieved by applying SemCL to a standard SEQ2SEQ model, demonstrating that the training objective can be more impactful than increasing model complexity.

Key Insights

  • The "Vague" Middle Ground: Handling Rank 1 samples (paraphrases) is critical. Treating them as strictly positive or negative degrades performance; acknowledging their ambiguity through ranked loss is the key to robustness.
  • Holistic Learning: By comparing entire utterance representations against entire MR representations, the model learns "global" meaning that token-by-token MLE misses.
  • Hard Negatives: "Online sampling"—using the model's own high-confidence errors as negative examples—is far more effective than random sampling for teaching fine-grained logical differences.

Conclusion and Future Work

SemCL effectively addresses the semantic limitations of MLE-based parsing. While it introduces additional computational overhead due to online sampling, the performance gains are substantial. Future research will focus on applying these contrastive techniques to low-resource scenarios where annotated data is scarce, potentially through unsupervised sampling methods.

Summary updated

Abstract

Since the meaning representations are detailed and accurate annotations which express fine-grained sequence-level semtantics, it is usually hard to train discriminative semantic parsers via Maximum Likelihood Estimation (MLE) in an autoregressive fashion. In this paper, we propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations and take the overall sequence-level semantic into consideration. Specifically, a multi-level online sampling algorithm is proposed to sample confusing and diverse instances. Three semantic-aware similarity functions are designed to accurately measure the distance between meaning representations as a whole. And a ranked contrastive loss is proposed to pull the representations of the semantic-identical instances together and push negative instances away. Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines and gets state-of-the-art performances by simply applying semantic-aware contrastive learning on a vanilla Seq2Seq model.
BibTeX
@inproceedings{wu2022semantic,
  author = {Shan Wu and Chunlei Xin and Bo Chen and Xianpei Han and Le Sun},
  title = {{Semantic-aware Contrastive Learning for More Accurate Semantic Parsing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.269},
}
Figure