Zipf’s law holds for phrases, not words
Name
Williams-2015-Zipf's law.pdf
Size
1.38 MB
Format
Adobe PDF
Checksum (MD5)
ccfca108c27197165f4974e16c5a1da9
Author(s)
Ryland Williams, Jake
Lessard, Paul R.
Desu, Suma
Clark, Eric M.
Bagrow, James P.
Danforth, Christopher M.
Sheridan Dodds, Peter
Date Issued
August 2015
Journal
Scientific Reports
Publisher
Nature Publishing Group
Citation
Ryland Williams, Jake, Paul R. Lessard, Suma Desu, Eric M. Clark, James P. Bagrow, Christopher M. Danforth, and Peter Sheridan Dodds. “Zipf’s Law Holds for Phrases, Not Words.” Scientific Reports 5 (August 11, 2015): 12209.
Version
Final published version
Abstract
With Zipf’s law being originally and most famously observed for word frequency, it is surprisingly limited in its applicability to human language, holding over no more than three to four orders of magnitude before hitting a clear break in scaling. Here, building on the simple observation that phrases of one or more words comprise the most coherent units of meaning in language, we show empirically that Zipf’s law for phrases extends over as many as nine orders of rank magnitude. In doing so, we develop a principled and scalable statistical mechanical method of random text partitioning, which opens up a rich frontier of rigorous text analysis via a rank ordering of mixed length phrases.
MIT Department
Massachusetts Institute of Technology. Center for Computational Engineering
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1038/srep12209
