Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts

Zipf’s law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf’s law should hold in the most natural way, studying its validity for plain word forms and for the corr...

Descripción completa

Detalles Bibliográficos
Autores: Corral, A., Boleda, G., Ferrer-i-Cancho, R.
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2015
País:España
Institución:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
Repositorio:Recercat. Dipósit de la Recerca de Catalunya
OAI Identifier:oai:recercat.cat:2072/377755
Acceso en línea:http://hdl.handle.net/2072/377755
Access Level:acceso abierto
Palabra clave:Matemàtiques
51
id ES_c4ff7cd42c2ca03ceb5a1eabaa0fbe12
oai_identifier_str oai:recercat.cat:2072/377755
network_acronym_str ES
network_name_str España
repository_id_str
spelling Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long TextsCorral, A.Boleda, G.Ferrer-i-Cancho, R.Matemàtiques51Zipf’s law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf’s law should hold in the most natural way, studying its validity for plain word forms and for the corresponding lemma forms. We analyze several long literary texts comprising four languages, with different levels of morphological complexity. In all cases Zipf’s law is fulfilled, in the sense that a power-law distribution of word or lemma frequencies is valid for several orders of magnitude. We investigate the extent to which the word-lemma transformation preserves two parameters of Zipf’s law: the exponent and the low-frequency cut-off. We are not able to demonstrate a strict invariance of the tail, as for a few texts both exponents deviate significantly, but we conclude that the exponents are very similar, despite the remarkable transformation that going from words to lemmas represents, considerably affecting all ranges of frequencies. In contrast, the low-frequency cut-offs are less stable, tending to increase substantially after the transformation.2015info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersion23 p.application/pdfhttp://hdl.handle.net/2072/377755RECERCAT (Dipòsit de la Recerca de Catalunya)reponame:Recercat. Dipósit de la Recerca de Catalunyainstname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)InglésPLoS ONEL'accés als continguts d'aquest document queda condicionat a l'acceptació de les condicions d'ús establertes per la següent llicència Creative Commons:http://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:recercat.cat:2072/3777552026-05-29T05:05:01Z
dc.title.none.fl_str_mv Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
title Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
spellingShingle Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
Corral, A.
Matemàtiques
51
title_short Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
title_full Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
title_fullStr Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
title_full_unstemmed Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
title_sort Zipf’s Law for Word Frequencies: Word Forms versus Lemmas in Long Texts
dc.creator.none.fl_str_mv Corral, A.
Boleda, G.
Ferrer-i-Cancho, R.
author Corral, A.
author_facet Corral, A.
Boleda, G.
Ferrer-i-Cancho, R.
author_role author
author2 Boleda, G.
Ferrer-i-Cancho, R.
author2_role author
author
dc.subject.none.fl_str_mv Matemàtiques
51
topic Matemàtiques
51
description Zipf’s law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf’s law should hold in the most natural way, studying its validity for plain word forms and for the corresponding lemma forms. We analyze several long literary texts comprising four languages, with different levels of morphological complexity. In all cases Zipf’s law is fulfilled, in the sense that a power-law distribution of word or lemma frequencies is valid for several orders of magnitude. We investigate the extent to which the word-lemma transformation preserves two parameters of Zipf’s law: the exponent and the low-frequency cut-off. We are not able to demonstrate a strict invariance of the tail, as for a few texts both exponents deviate significantly, but we conclude that the exponents are very similar, despite the remarkable transformation that going from words to lemmas represents, considerably affecting all ranges of frequencies. In contrast, the low-frequency cut-offs are less stable, tending to increase substantially after the transformation.
publishDate 2015
dc.date.none.fl_str_mv 2015
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/2072/377755
url http://hdl.handle.net/2072/377755
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv PLoS ONE
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv 23 p.
application/pdf
dc.source.none.fl_str_mv RECERCAT (Dipòsit de la Recerca de Catalunya)
reponame:Recercat. Dipósit de la Recerca de Catalunya
instname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
instname_str Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
reponame_str Recercat. Dipósit de la Recerca de Catalunya
collection Recercat. Dipósit de la Recerca de Catalunya
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869418950555598848
score 15,811543

Similares en LA Referencia