Automatic Analysis of Hungarian Texts and Linguistic Data

Ferenc PappCOLING 1973

TL;DR

This paper details an early attempt at automatically analyzing Hungarian texts by processing a large dictionary and natural language corpora. The authors developed methods to extract and analyze various linguistic features, demonstrating how computational approaches can reveal insights into language structure, historical changes, and even stylistic nuances.

What Problem Does This Paper Address?

The paper addresses the challenge of computationally analyzing and understanding natural language, specifically Hungarian. Traditional linguistic analysis is time-consuming and often subjective. The authors sought to automate this process by leveraging computational power to extract and analyze a wide range of linguistic data from dictionaries and texts, aiming to uncover patterns and answer linguistic questions that would be difficult or impossible to address manually. This includes understanding word structure, grammatical properties, and even historical linguistic evolution.

Methods

The core of the methodology involves processing linguistic data using computational techniques. The authors first processed a large Hungarian dictionary, extracting and coding various linguistic features for each headword. These features included:

  • Lexical complexity: Number of roots in a lexeme.
  • Homonymy: Identifying words with multiple meanings.
  • Part of speech: Categorizing words and their potential conversions.
  • Number of meanings: Quantifying polysemy.
  • Style: Classifying word usage.
  • Morphological data: Information for inflected words.
  • Government: For verbs, identifying required grammatical complements.
  • Etymology: Word origins.
  • Suffixes: Identifying word-forming suffixes.

Beyond dictionary processing, the paper describes preparing natural Hungarian texts for automatic analysis. This involved developing programs to:

  • Convert orthography to phonemes: Translating written text into its sound representation.
  • Identify and count phonemes: Analyzing the frequency and distribution of sounds.
  • Multi-dimensional space representation: Placing texts in a spatial model based on phoneme statistics to measure distances and relationships between them.

Datasets

  • Hungarian Unilingual (Explanatory) Dictionary: Approximately 60,000 entries from A magyar nyelv értelmező szótára (Volumes I-VII, 1959-1962). This served as the primary source for detailed linguistic feature extraction.
  • Natural Hungarian Texts: A corpus of half a million phonemes, specifically the complete poetic works of Endre Ady, was used for automatic processing. The authors planned to expand this with other 20th-century and earlier Hungarian poets.

Key Insights and Ideas

  • Computational linguistics can reveal hidden linguistic patterns: By systematically processing large amounts of data, computational methods can uncover trends and relationships that are not easily discernible through manual analysis.
  • Linguistic features are quantifiable: The paper demonstrates that various linguistic aspects, from word complexity to etymological strata, can be represented and analyzed numerically.
  • Vowel harmony's historical evolution: The analysis of etymological strata revealed a shift towards more velar elements over time, with Latin loanwords being predominantly velar. This provides a quantitative perspective on language change.
  • Vowel proportion in texts is linked to word length: Contrary to a common belief that texts with more vowels are inherently more "beautiful," the study found that vowel proportion correlates with the average length of running words, with longer words being more vocalic.
  • Stylistic differentiation through phoneme statistics: The multi-dimensional space analysis of texts showed that even early works of a poet can be acoustically distinct from their later oeuvre, suggesting that phoneme statistics can capture stylistic differences.

Key Results

  • The processing of the dictionary yielded structured linguistic data for each headword, enabling various combinations and queries.
  • Analysis of the Ady corpus revealed that poetic texts with shorter words tend to be less vocalic than technical texts.
  • The multi-dimensional space analysis of Ady's works showed that his first juvenile volume was acoustically distant from his later works, indicating a significant stylistic shift.
  • The study provided quantitative indices for distances between Finno-Ugrian and Neo-Latin etymological layers, suggesting potential applications in other fields.

Limitations and Future Work

  • Dictionary size and its impact: The authors acknowledge that the distribution of certain data, like the number of meanings per lexeme, might be influenced by the size of the dictionary and could differ in larger corpora.
  • Critical view of length data: Data on lexeme length was noted as requiring a critical view, as larger dictionaries might contain more longer words.
  • Future work: The authors envisioned expanding this work to include sound statistics, poetic concordance dictionaries, and morphological and syntactical processing of the material. They also suggested that the developed models could be applied to other fields.

Summary updated

BibTeX
@inproceedings{papp1973automatic,
  author = {Ferenc Papp},
  title = {{Automatic Analysis of Hungarian Texts and Linguistic Data}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1973},
}
Figure