On the Power Laws of Language: Word Frequency Distributions
Lists
Only you see this
Abstract
About eight decades ago, Zipf postulated that the word frequency distribution of languages is a power law, i.e., it is a straight line on a log-log plot. Over the years, this phenomenon has been documented and studied extensively. For many corpora, however, the empirical distribution barely resembles a power law: when plotted on a log-log scale, the distribution is concave and appears to be composed of two differently sloped straight lines joined by a smooth curve. A simple generative model is proposed to capture this phenomenon. The word frequency distributions produced by this model are shown to match the observations both analytically and empirically.
BibTeX
@inproceedings{chierichetti2017power,
author = {Flavio Chierichetti and Ravi Kumar and Bo Pang},
title = {{On the Power Laws of Language: Word Frequency Distributions}},
booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2017},
doi = {10.1145/3077136.3080821},
}Loading related content...