Applying automatically parsed corpora to the study of language variation

Jelke Bloem, Arjen Versloot, Fred WeermanCOLING 2014

TL;DR

This paper demonstrates the power of automatically parsed corpora for studying language variation, showing that large-scale, automated annotation can replicate and extend previous manual studies. By applying this method to Dutch verbal cluster order, the authors confirm existing findings and reveal that these patterns hold across different text domains, highlighting the flexibility and scalability of automated linguistic analysis.

What Problem Does This Paper Address?

Linguistic variation, the phenomenon of having multiple ways to express the same meaning, is a rich area of study. Traditionally, researchers have used manual or semi-automatic annotation of corpora to identify and analyze these variations. However, this process is time-consuming and limits the size of datasets that can be analyzed, potentially hindering the development of complex statistical models needed to capture nuanced linguistic phenomena. This paper addresses the challenge of scaling up quantitative studies of language variation by leveraging automatically parsed corpora, which can provide vastly larger datasets and enable more robust analysis. Specifically, the authors focus on the variation in word order within Dutch verbal clusters, a phenomenon that has been extensively studied but could benefit from larger-scale, automated investigation.

Methods

The core of this paper's methodology lies in the application of automatically parsed corpora to linguistic variation. The authors utilize the Lassy Large corpus, which contains Dutch texts parsed with the Alpino parser, to extract verbal clusters. They define specific criteria for identifying these clusters (e.g., two adjacent verbs in a subordinate clause with one as the head of the other) and use XPath queries to automatically extract them.

The study then employs multivariate logistic regression, a standard statistical technique in quantitative linguistics, to model the factors influencing verbal cluster order. Crucially, they aim to replicate and extend the work of De Sutter (2009), who used a manually annotated corpus. This involves operationalizing variables identified in previous literature (e.g., morphological structure of the main verb, length of the middle field, information value of the preverbal constituent) using the information available in the automatic parse trees. They also extend the analysis by including cluster types (modal, infinitival, main clause) that were excluded in De Sutter's original study. To ensure comparability, they largely adhere to De Sutter's variable operationalizations and reference values, while also acknowledging where differences were necessary due to the nature of automatic annotation.

Datasets

The primary dataset used in this study is the Lassy Large corpus, a collection of automatically annotated Dutch language data. For their main analysis, the authors focus on the Wikipedia part of this corpus, comprising approximately 145 million words. They extracted 411,623 two-verb verbal clusters from this subset.

For a comparative analysis and to test domain specificity, they also utilize the Europarl part of the Lassy Large corpus, which contains translated proceedings of the European Parliament. This subset consists of 37 million words and yielded 467,521 verbal clusters.

The study does not introduce new benchmarks but rather aims to replicate and extend existing work on a well-studied phenomenon using a different type of data source.

Key Insights and Ideas

  • Automated annotation unlocks scale: The most significant insight is that automatically parsed corpora, despite potential parsing errors, provide a massive dataset that can overcome the limitations of manual annotation, enabling more comprehensive studies of language variation.
  • Generalizability of variation factors: The study demonstrates that the linguistic variables identified as influential in previous, smaller-scale studies of Dutch verbal clusters are also significant in much larger and more diverse datasets, and importantly, across different text domains (Wikipedia vs. Europarl).
  • Flexibility of automated methods: The ease with which the authors could extend their analysis to include previously excluded cluster types (modal, infinitival, main clause) highlights the flexibility of using automated parsing. This allows for a more holistic view of the phenomenon.
  • Confirmation and extension of prior work: The study successfully replicates the core findings of De Sutter (2009) and extends them to a broader range of verbal cluster constructions, reinforcing the robustness of the identified explanatory variables.
  • Domain independence: The similar results obtained from the Wikipedia and Europarl corpora suggest that the factors influencing verbal cluster order are not specific to a particular genre or register, pointing to underlying linguistic principles.

Key Results

  • The multivariate model, using automatically annotated data, achieved a concordance index of 0.8635, indicating strong predictive power, comparable to or slightly better than De Sutter's manually annotated model (c=0.803).
  • The study confirmed that seven of the eight variables identified by De Sutter (2009) were significant predictors of verbal cluster order.
  • The TYPE OF AUXILIARY variable showed a particularly strong effect, with modal verbs exhibiting a strong preference for the 1-2 order (odds ratio of 132.42).
  • The INFORMATION VALUE of the preverbal constituent showed an effect in the same direction as De Sutter's study but was smaller, potentially due to the larger sample size and inclusion of additional cluster types.
  • Analysis of the Europarl corpus showed a higher percentage of 1-2 orders (86.78%) compared to Wikipedia (71.65%), aligning with the expectation that more formal registers favor this order. Crucially, the effect directions of the variables remained largely consistent across both domains.

Limitations and Future Work

  • Annotation limitations: The study acknowledges that the available automatic annotation limits the operationalization of certain variables. For instance, variables related to stress patterns (DISTANCE BETWEEN ACCENTS) or syntactic priming (SYNTACTIC PERSISTENCE) could not be included.
  • Parser accuracy: While the Lassy Large corpus boasts high accuracy, systematic parsing errors could still skew results, particularly for constructions the parser consistently fails to annotate.
  • Domain specificity vs. generalizability: While the study demonstrates domain independence for the tested variables, it notes that regional variation, which has been observed in Dutch verbal clusters, could be further explored using corpora with author metadata.
  • Future work suggestions:
    • Extending the model to other domains, such as spoken language data.
  • Investigating clusters with more than two verbs, where automated methods are particularly suited due to their rarity.
  • Exploring collostructional analysis to uncover associations between specific main verbs and word order.
  • Aiming for the "best possible model" rather than strict comparability, potentially using multilevel modeling or Principal Component Analysis.
  • Investigating syntactic complexity as a variable related to processing difficulty.

Summary updated

BibTeX
@inproceedings{bloem2014applying,
  author = {Jelke Bloem and Arjen Versloot and Fred Weerman},
  title = {{Applying automatically parsed corpora to the study of language variation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2014},
}
Figure