The variation of Zipf's law in human language
Words in humans follow the so-called Zipf's law. More precisely, the word frequency spectrum follows a power function, whose typical exponent is ß ˜ 2, but significant variations are found. We hypothesize that the full range of variation reflects our ability to balance the goal of communication...
| Autor: | |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2005 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/180136 |
| Acceso en línea: | https://hdl.handle.net/2117/180136 https://dx.doi.org/10.1140/epjb/e2005-00121-8 |
| Access Level: | acceso abierto |
| Palabra clave: | Information theory Computational linguistics Zipf’s law Large-scale systems Linguistics Natural languages Probability Speech Informació, Teoria de la Lingüística computacional Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural |
| Sumario: | Words in humans follow the so-called Zipf's law. More precisely, the word frequency spectrum follows a power function, whose typical exponent is ß ˜ 2, but significant variations are found. We hypothesize that the full range of variation reflects our ability to balance the goal of communication, i.e. maximizing the information transfer and the cost of communication, imposed by the limitations of the human brain. We show that the higher the importance of satisfying the goal of communication, the higher the exponent. Here, assuming that words are used according to their meaning we explain why variation in ß should be limited to a particular domain. From the one hand, we explain a non-trivial lower bound at about ß = 1.6 for communication systems neglecting the goal of the communication. From the other hand, we find a sudden divergence of ß if a certain critical balance is crossed. At the same time a sharp transition to maximum information transfer and unfortunately, maximum communication cost, is found. Consistently with the upper bound of real exponents, the maximum finite value predicted is about ß = 2.4. It is convenient, for human language not to cross the transition and remain in a domain where maximum information transfer is high but at a reasonable cost. Therefore, only a particular range of exponents should be found in human speakers. The exponent ß contains information about the balance between cost and communicative efficiency. |
|---|