Convex Polytope Modelling for Unsupervised Derivation of Semantic Structure for Data-efficient Natural Language Understanding

Jingyan Zhou, Xiaohan Feng, King Keung Wu, Helen MengarXiv 2022 · arXiv

Abstract

Natural Language Understanding (NLU) for task-oriented dialog systems typically requires extensive labeled data to map user utterances to semantic frames. This paper introduces the Convex Polytopic Model (CPM), an unsupervised geometric framework that extracts semantic patterns from raw data by representing utterances within a minimum volume simplex. The vertices of this polytope serve as semantic archetypes, providing a structured prior that enhances NLU performance. Experimental results on the ATIS corpus demonstrate that CPM-guided models significantly outperform baselines in low-data scenarios, achieving a 26% relative improvement in F1 score with only 2.5% of labeled training data.

Introduction

The development of robust task-oriented dialog systems is often hindered by the "data hunger" of supervised NLU models. Mapping a user's natural language query into a structured format—consisting of an intent and various slots—traditionally requires thousands of manually labeled examples. Furthermore, deep learning models used for this task are frequently criticized for their lack of interpretability. This research proposes a geometric approach to discover the underlying semantic structure of a domain without human intervention, using this structure to guide supervised learning and improve data efficiency.

The NLU Task and Semantic Frames

The primary objective of NLU in this context is to fill a "semantic frame." For instance, a query about air travel must be parsed into specific components that a database can understand.

Example of a semantic frame mapping an utterance to intents and slots.

As shown in the figure above, the system must identify the core intent (e.g., FLIGHTS) and extract relevant entities or "slots" (e.g., ORIGIN, DESTINATION, DATE). Traditionally, this mapping is performed via handcrafted rules or supervised learning, both of which have significant scaling and transparency limitations.

Methodology: The Convex Polytopic Model

The authors propose a three-stage pipeline to derive semantic features from unlabeled data:

  1. Vectorization: Utterances are converted into bag-of-words vectors, normalized to lie on a high-dimensional hyperplane.
  2. Dimensionality Reduction: Principal Component Analysis (PCA) is applied to project the data into a lower-dimensional affine subspace, capturing the most significant semantic variances.
  3. Minimum Volume Simplex (MVS): The core of the CPM involves finding a polytope that tightly encloses the projected points. The vertices of this polytope represent "pure" semantic archetypes. Any given utterance is then represented as a convex combination (a weighted mixture) of these archetypes.

To integrate these geometric insights into a neural architecture, the researchers developed a CPM-guided attention mechanism. By calculating a word-word correlation matrix based on the unsupervised geometry, they provide the model with a "semantic map" that helps it focus on relevant tokens during the slot-filling process.

Experimental Results

The framework was evaluated using the ATIS (Air Travel Information System) dataset, a standard benchmark for flight-related queries. The researchers focused on the model's performance across varying amounts of labeled training data to test data efficiency.

The results highlight a dramatic increase in performance within the "small data" regime. When using only 2.5% of the training set, the CPM-guided model achieved an F1 score of 66.07%, whereas the baseline model reached only 52.39%. This represents a relative improvement of over 26%. Even as the volume of training data increased to 90%, the CPM-guided approach maintained a slight edge (94.60% vs 94.24%), suggesting that the geometric prior provides a cleaner, more reliable signal than learned attention alone.

Interpretability and Visualization

Beyond performance gains, the CPM offers inherent interpretability. Because the model is grounded in geometry, the relationship between words and semantic slots can be visualized intuitively. The authors utilized "schema-balls" to map the discovered unsupervised vertices to known slots, providing a transparent view of what the model has learned. This geometric approach allows researchers to see exactly which words the model associates with specific concepts like TIME or ORIGIN based purely on the data's underlying structure.

Conclusion

Convex Polytope Modelling offers a powerful, unsupervised method for discovering semantic structure in conversational data. By leveraging the geometric properties of language, the CPM framework reduces the reliance on large labeled datasets and provides a more interpretable foundation for Natural Language Understanding in task-oriented systems.

Summary updated

BibTeX
@misc{zhou2022convex,
  author = {Jingyan Zhou and Xiaohan Feng and King Keung Wu and Helen Meng},
  title = {{Convex Polytope Modelling for Unsupervised Derivation of Semantic Structure for Data-efficient Natural Language Understanding}},
  year = {2022},
  eprint = {2201.10588},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.10588},
}
Figure