Comparing Symbolic Models of Language via Bayesian Inference (Student Abstract)

Annika Heuser, Polina TsvilodubAAAI 2021 · DOI

Abstract

This paper addresses the challenge of comparing symbolic language models that parse different, noisy subsets of a dataset. By extending Bayesian scoring methods to account for varying coverage and sentence length, the authors evaluate four grammatical hypotheses for German word-order acquisition. The results demonstrate that a hierarchical grammar (SOV+V2) is preferred by a Bayesian learner, even when faced with significant noise in standard linguistic datasets.

Introduction

A fundamental question in linguistics and artificial intelligence is how children acquire complex language rules from limited and often noisy input—the "poverty of the stimulus" problem. While modern neural networks rely on vast amounts of data, humans exhibit a remarkable ability to generalize from sparse examples. This has led researchers to hypothesize an innate "initial bias" toward symbolic, hierarchical representations.

Previous work, such as Perfors et al. (2011), utilized Bayesian inference to show that learners naturally favor hierarchical grammars over linear ones. However, these studies often assumed that models could parse the entire dataset perfectly. In practice, real-world language data is messy. If one grammar parses a larger but noisier subset than another, standard Bayesian comparison becomes problematic. This research bridges this gap by introducing a scoring procedure that allows for the fair evaluation of competing symbolic models supported by different subsets of noisy data.

Methodology

The authors developed four Probabilistic Context-Free Grammars (PCFGs) representing different hypotheses for German word order:

  1. SVO: Subject-Verb-Object
  2. SOV: Subject-Object-Verb
  3. SVO+V2: SVO with "Verb-Second" rules
  4. SOV+V2: SOV with "Verb-Second" rules (the closest approximation to standard German)

To compare these models fairly, a new scoring procedure was introduced to handle the discrepancies in data coverage. The procedure includes:

  • Length Normalization: Calculating average sentence log-likelihood normalized by sentence length to avoid penalizing models for parsing complex structures.
  • Coverage Weighting: Weighing the normalized log-likelihood by the proportion of unique sentence types the grammar successfully parses.

Data and Preprocessing

The study utilized the Leo corpus from the CHILDES dataset, a standard repository for child-directed speech. Sentences were converted into Part-of-Speech (POS) tags using the STTS tag set. After manual filtering to remove fillers and ungrammatical fragments, 160,635 utterances remained.

A significant finding during preprocessing was the high level of noise. Approximately 31.65% of the corpus consisted of ungrammatical strings or tagging errors, highlighting a potential issue with "gold standard" datasets in language acquisition research.

Results

The application of the adjusted Bayesian scoring procedure revealed a clear preference for the hierarchical SOV+V2 grammar. This model aligns with the actual linguistic structure of German.

Table 1: Bayesian scores for the four grammatical hypotheses. The SOV+V2 model shows the highest log-likelihoods, indicating it is the preferred model.

As shown in Figure 1, the SOV+V2 grammar achieves the highest (least negative) log-probabilities across the key metrics, including the posterior $P(G, T|D)$. Despite its superior performance, the SOV+V2 grammar could only parse 48.98% of the preprocessed dataset, reflecting the inherent noise in the input. Manual inspection of a sample of 200 strings yielded a sensitivity of 0.808 and a specificity of 0.802, confirming the grammar's effectiveness in distinguishing grammatical German from noise.

Discussion and Conclusion

The results support the hypothesis that Bayesian learners possess a bias toward hierarchical structures. By normalizing for length and coverage, the authors demonstrate that Bayesian inference can be a robust tool for model selection even in "the wild," where data is imperfect.

However, the study also serves as a cautionary note regarding data quality. If nearly a third of a corpus is "garbage," it complicates conclusions about how children develop perfect internal grammars from such input. Future research will focus on scaling this method to more complex syntactic components, improving data tagging and preprocessing, and expanding the study to other languages to test the universality of the hierarchical preference.

Summary updated

BibTeX
@inproceedings{heuser2021comparing,
  author = {Annika Heuser and Polina Tsvilodub},
  title = {{Comparing Symbolic Models of Language via Bayesian Inference (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17896},
}
Figure