On the Power Laws of Language: Word Frequency Distributions

Flavio Chierichetti, Ravi Kumar, Bo Pang

Abstract

About eight decades ago, Zipf postulated that the word frequency distribution of languages is a power law, i.e., it is a straight line on a log-log plot. Over the years, this phenomenon has been documented and studied extensively. For many corpora, however, the empirical distribution barely resembles a power law: when plotted on a log-log scale, the distribution is concave and appears to be composed of two differently sloped straight lines joined by a smooth curve. A simple generative model is proposed to capture this phenomenon. The word frequency distributions produced by this model are shown to match the observations both analytically and empirically.
BibTeX
@inproceedings{chierichetti2017power,
  author = {Flavio Chierichetti and Ravi Kumar and Bo Pang},
  title = {{On the Power Laws of Language: Word Frequency Distributions}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2017},
  doi = {10.1145/3077136.3080821},
}

Loading related content...

Figure