A Computational Model of Language Performance: Data Oriented Parsing

Rens BodCOLING 1992

TL;DR

This paper introduces Data Oriented Parsing (DOP), a novel approach to natural language processing that models language use based on experience (an analyzed corpus) rather than abstract rules. DOP aims to bridge the gap between linguistic competence and actual language performance by finding the most probable way to reconstruct input sentences from existing fragments in a corpus, naturally handling ambiguity as a side-effect.

What Problem Does This Paper Address?

Traditional natural language processing (NLP) models often rely on "competence grammars," which aim to capture the abstract, idealized knowledge of language. However, these models struggle with the inherent ambiguity of natural language, leading to an explosion of possible interpretations for even simple sentences. Human language users, on the other hand, exhibit strong preferences for certain interpretations, suggesting that actual language processing is more about "performance" – how language is actually used and experienced. The paper argues that existing statistical approaches, while useful for some tasks, often ignore crucial syntactic structure, and that a more integrated approach is needed to capture both the statistical regularities and the structural complexity of language.

Methods

The core of DOP is to treat an analyzed corpus of sentences and their syntactic structures as the basis for language processing. Instead of applying grammatical rules, DOP attempts to find the most probable way to reconstruct an input sentence by combining "constructions" (fragments of trees representing syntactic structures) extracted from the corpus. This process is framed as finding an optimal analogy between the input sentence and the corpus.

Key methodological aspects include:

  • Corpus as Experience: The system learns from a large, annotated corpus, where each sentence is represented as a tree structure.
  • Constructions: These are subtrees or patterns abstracted from the corpus trees, representing recurring linguistic units.
  • Composition: Trees (constructions) are combined using a composition operation (denoted by 'o') to build up parses for new sentences.
  • Probabilistic Parsing: The "preferred parse" for an input sentence is the one that can be generated by the most probable combination of constructions from the corpus. This probability is derived from the frequency of these constructions in the corpus.
  • Disambiguation as a Side-Effect: By favoring more frequent constructions and combinations, DOP naturally resolves ambiguities, mirroring human preferences for more plausible interpretations.

The paper also touches upon implementation strategies, suggesting that conventional rule-based parsing techniques can be adapted by treating constructions as production rules. For efficiency, Monte Carlo techniques are proposed to estimate the preferred parse rather than exhaustively calculating all possibilities.

Datasets

The paper does not explicitly list specific datasets used for training or evaluation. However, it refers to the necessity of a "very large corpus of sentences with their syntactic analyses and semantic interpretations." It also mentions potential corpora for testing, such as the Tosca Corpus and the Penn Treebank.

Key Insights and Ideas

  • Performance over Competence: The paper strongly advocates for modeling language performance, which is grounded in actual language use and experience, rather than abstract competence.
  • Corpus as the Primary Source: Language understanding is framed as an act of analogy and reconstruction based on past linguistic experiences stored in a corpus.
  • Statistical Regularities and Structural Complexity: DOP aims to unify statistical information (frequencies of constructions) with structural information (tree fragments), addressing a limitation of purely statistical or purely rule-based systems.
  • Natural Handling of Ambiguity: Ambiguity is not treated as a problem to be solved by explicit rules but emerges as a natural consequence of favoring more frequent interpretations derived from the corpus.
  • Flexibility in Construction Granularity: The system can utilize constructions of varying sizes, from single words to larger phrasal units, allowing it to capture both local and long-distance dependencies.

Key Results

The paper is primarily theoretical and introduces the Data Oriented Parsing (DOP) model. It does not present quantitative results from experiments. The "key results" are conceptual:

  • A formalization of Data Oriented Parsing.
  • A demonstration of how DOP can handle ambiguity by favoring more probable interpretations derived from corpus statistics.
  • A proposal for how DOP can be implemented using existing parsing strategies.
  • An argument for the adequacy of DOP in processing phenomena like prepositional phrase attachments and figures of speech due to its ability to leverage larger constructions.

Limitations and Future Work

  • Computational Inefficiency: The paper acknowledges that calculating the preferred parse by generating all possible parses and their construction tuples is "highly inefficient." Monte Carlo methods are suggested as a way to mitigate this.
  • Language Acquisition: The authors note that using an already analyzed corpus might seem to limit the model to simulating adult language behavior. They suggest that language acquisition itself involves acquiring an initial corpus, where non-linguistic input and pragmatics play a role.
  • Testing and Evaluation: The paper states that DOP "has not yet been tested thoroughly." Future work would involve extensive testing on various corpora and tasks.
  • Corpus Size and Quality: The effectiveness of DOP is inherently tied to the size and quality of the analyzed corpus. The paper implies that a comprehensive corpus is crucial.

Summary updated

BibTeX
@inproceedings{bod1992computational,
  author = {Rens Bod},
  title = {{A Computational Model of Language Performance: Data Oriented Parsing}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1992},
}
Figure