TL;DR
This paper investigates Zipf's Law, which states word frequency is inversely proportional to rank, and finds it holds for single words only at high frequencies. However, by combining single words with n-gram phrases, the authors demonstrate that Zipf's Law can be extended to accurately describe word and phrase frequencies across all ranks in both English and Mandarin, suggesting a more universal linguistic principle.
What Problem Does This Paper Address?
Zipf's Law, a well-established empirical observation in linguistics, posits that the frequency of a word in a corpus is inversely proportional to its rank. While this law has been widely studied and applied, its validity has been questioned, particularly for larger corpora. Previous research indicated that for corpora exceeding a certain size (around 5,000 words), the Zipf curve deviates from the expected straight line with a slope of -1 at lower frequencies. This paper aims to explore this deviation, particularly in the context of large corpora and different languages (English and Mandarin), and to investigate whether incorporating multi-word units (n-grams) can restore the law's universality.
Methods
The core methodology involves analyzing word and n-gram frequencies in large text corpora. The authors extract unigrams (single words) and n-grams (sequences of 2 to 5 words) from their datasets. They then calculate the frequency of each unit and rank them accordingly. Zipf curves are generated by plotting the logarithm of frequency against the logarithm of rank. To assess the deviation from Zipf's Law, they examine the slope of these curves, particularly at lower ranks. A key novel aspect is the combination of unigram and n-gram frequencies into a single list to create a "combined n-gram" Zipf curve, hypothesizing that this unified approach might better reflect the law's behavior.
Datasets
The paper utilizes the following datasets:
- English Corpora:
- The Wall Street Journal (WSJ) corpus, split into three parts for 1987, 1988, and 1989, with sizes approximately 19 million, 16 million, and 6 million tokens, respectively. A combined WSJ corpus of 40 million words was also used for comparisons.
- The Brown corpus, a 1-million-word corpus of American English.
- Mandarin Corpus:
- The TREC Corpus, obtained from the People's Daily Newspaper (01/1991 to 12/1993) and the Xinhua News Agency (04/1994 to 09/1995). This corpus contains 19,546,872 tokens.
Key Insights and Ideas
- Limited Scope of Unigram Zipf's Law: The paper confirms that Zipf's Law for single words (unigrams) is only accurate for high-frequency words in large corpora. At lower frequencies, the observed frequencies fall below the predicted line.
- N-grams Restore Universality: A significant finding is that when n-grams (phrases of 2 to 5 words) are combined with unigrams, the resulting "combined n-gram" Zipf curve accurately follows the law across all ranks, for both English and Mandarin. This suggests that Zipf's Law might be more fundamentally about the distribution of meaningful units in language, whether single words or phrases.
- Language-Agnostic Principle: The fact that the combined n-gram Zipf curves for English and Mandarin are nearly identical strongly suggests that Zipf's Law, in this extended form, represents a language-agnostic principle of linguistic organization.
- Simon's Derivation Revisited: The authors suggest that their findings vindicate Simon's derivation of Zipf's Law, which they argue implicitly should have included phrases, not just single words.
Key Results
- For large English corpora (e.g., 1 million words), Zipf curves for unigrams deviate from the straight line (slope -1) at ranks greater than approximately 5,000.
- The n-gram Zipf curves (2-grams to 5-grams) for both English and Mandarin corpora generally follow straight lines, but with slopes less than -1 (e.g., for WSJ, 2-grams have slopes around 0.65-0.67, and 5-grams around 0.41-0.42).
- The combined unigram and n-gram Zipf curves for both English and Mandarin are straight lines with slopes close to -1 for ranks greater than 100.
- The Mandarin unigram Zipf curve is significantly different from the English unigram curve due to the much smaller number of characters (syllable-types) in Mandarin compared to English word types. However, the n-gram curves for Mandarin are more curved than for English, attributed to the prevalence of compound words.
Limitations and Future Work
- Limitations: The paper acknowledges that the original Zipf's Law derivation by Simon was based solely on single words and was controversial. While their combined n-gram approach restores the law's validity, the initial deviation of unigrams in large corpora remains a point of discussion. The authors also note that the Mandarin n-gram curves are more curved than English ones, suggesting that the "compound word" nature of Mandarin might require further nuanced analysis.
- Future Work: The authors mention that preliminary experiments have shown this extended Zipf's Law to be valid for other natural languages, including Irish, Latin, and Vietnamese. This suggests a promising avenue for future research to explore the universality of this combined n-gram Zipf's Law across a wider range of languages and to further investigate the linguistic underpinnings of this phenomenon.
Summary updated