A Compositional Bayesian Semantics for Natural Language

Jean-Philippe Bernardy, Rasmus Blanck, Stergios Chatzikyriakidis, Shalom LappinCOLING 2018

TL;DR

This paper introduces a novel compositional Bayesian semantics for natural language, treating sentence interpretation as probabilistic inference. By representing linguistic elements as vectors in a probabilistic framework, it offers a principled way to handle semantic vagueness and learn linguistic patterns, moving beyond traditional truth-conditional semantics.

What Problem Does This Paper Address?

Traditional formal semantics, rooted in model theory, assigns Boolean truth values to sentences, which struggles to account for the inherent vagueness in natural language (e.g., "tall," "red") and the process of semantic learning. While recent probabilistic semantics approaches have emerged, they often suffer from shortcomings like lacking explicit treatments of vagueness or requiring extensive pre-defined knowledge. This paper aims to bridge this gap by proposing a compositional Bayesian semantics that naturally handles vagueness and supports semantic learning through probabilistic inference.

Methods

The core of the proposed semantics is a compositional Bayesian approach implemented in a functional programming language (Haskell) that leverages a probabilistic programming language (WebPPL) for inference. Key methodological aspects include:

  • Vector Space Models: Objects and properties are represented as vectors in multi-dimensional vector spaces.
  • Probabilistic Interpretation: Declarative sentences are assigned conditional probabilities, estimating the likelihood that an idealized speaker would endorse an assertion given certain hypotheses. This is framed as $P(A|H)$.
  • Markov Chain Monte Carlo (MCMC) Sampling: Used to estimate marginal probabilities of sentences by sampling from vector space models that satisfy specified hypotheses.
  • Compositionality: The semantics builds up sentence meaning from the meanings of its parts, following a Montague-style approach.
  • Probabilistic Programming: The system utilizes a probabilistic programming framework with an effect system to explicitly manage sampling and observations, making the probabilistic nature of computations clear.
  • Handling Vagueness: Vagueness is addressed by treating predicates as probabilistic measures with associated uncertainty (e.g., Gaussian error), allowing for fuzzy boundaries rather than strict cut-offs.
  • Generalised Quantifiers: These are interpreted probabilistically, moving beyond simple cardinality constraints to expected values of predicate satisfaction.
  • Semantic Learning: The framework supports learning by inferring correlations between properties (represented as vectors) based on observed data.

Datasets

The paper does not explicitly mention specific datasets for training or evaluation in the traditional machine learning sense. Instead, it uses:

  • Illustrative Examples: The paper relies on carefully constructed examples to demonstrate the semantics' capabilities, such as interpreting sentences involving predicates like "tall," "red," "bird," "animal," and "fly," as well as generalised quantifiers like "most" and "many."
  • Wikipedia Data: For the semantic learning section, data on person height, weight, and foot size was taken from a Wikipedia page (https://en.wikipedia.org/wiki/Naive_Bayes_classifier). This data is used to illustrate how the system can learn correlations between continuous variables.

Key Insights and Ideas

  • Vagueness as Intrinsic: The paper argues that vagueness is an intrinsic property of predication and does not require positing sharp, context-dependent boundaries for predicate application. This is a significant departure from some other probabilistic semantics approaches.
  • Unified Treatment of Predicates: The framework provides a consistent way to model both gradable (e.g., "tall") and non-gradable (e.g., "red") predicates, as well as generalised quantifiers, within a single probabilistic model.
  • Probabilistic Inference as Meaning: The core idea that sentence meaning is a probability distribution, rather than a set of truth conditions, allows for a more nuanced and flexible interpretation of language.
  • Learning through Correlation: Semantic learning is framed as learning correlations between vector representations of properties, which naturally extends to inferring relationships between linguistic concepts.
  • Computational Tractability: By using MCMC sampling within a probabilistic programming language, the system offers a computationally feasible way to estimate these probabilities, even for complex inferences.

Key Results

The paper presents qualitative results demonstrating the system's ability to:

  • Estimate Probabilities: Successfully compute approximate probabilities for various sentence types, including those involving vague predicates and generalised quantifiers. For instance, in the "bird example," the system concludes with high probability (0.941) that "most animals are not birds" given the premises.
  • Handle Comparatives: Infer that "John is tall" is slightly more probable (0.552) than not, given the observation "John is taller than Mary."
  • Model Vague Predicates: Show that "most animals are not birds" is highly probable (0.941) in a scenario where "most animals don't fly," "most birds fly," and "every bird is an animal."
  • Learn Correlations: Demonstrate learning a correlation between weight and maleness, predicting an individual of weight 1.9 is male with a probability of 0.57805. The posterior distribution also indicates a strong correlation.

Limitations and Future Work

Limitations:

  • Prototype System: The authors acknowledge that the implemented system is a prototype and a proof of concept. A robust, wide-coverage version is needed for broader applications.
  • Approximation: The use of MCMC sampling means that the results are approximations and can vary between runs.
  • Limited Syntactic Coverage: The paper focuses on specific sentence structures and does not cover the full complexity of natural language syntax.

Future Work:

  • Extend Syntactic and Semantic Coverage: Improve the framework to handle a wider range of linguistic phenomena.
  • Improve Efficiency and Robustness: Develop more efficient and robust modelling and sampling mechanisms for large-scale applications.
  • Advanced Bayesian Learning: Extend the Bayesian learning theory to handle more complex cases of classifier acquisition.
  • Integration with Real-World Knowledge: Encode semantic and real-world knowledge as priors to support more sophisticated inferencing and question answering.
  • Multimodal Integration: Envisage integrating visual and other non-linguistic vector representations.
  • Machine Translation Filter: Explore using the system as a filter for machine translation.

Summary updated

BibTeX
@inproceedings{bernardy2018compositional,
  author = {Jean-Philippe Bernardy and Rasmus Blanck and Stergios Chatzikyriakidis and Shalom Lappin},
  title = {{A Compositional Bayesian Semantics for Natural Language}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure