Selection of correction candidates for the normalization of Spanish user generated content

We present research aiming to build tools for the normalization of User-Generated Content (UGC). We argue that processing this type of text requires the revisiting of the initial steps of Natural Language Processing, since UGC (micro-blog, blog, and, generally, Web 2.0 user-generated texts) presents...

Full description

Bibliographic Details
Authors: Melero, Maite, Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X, Lambert, Patrik, Quixal, Martí
Format: article
Publication Date:2016
Country:España
Institution:Universitat Politècnica de Catalunya (UPC)
Repository:UPCommons. Portal del coneixement obert de la UPC
Language:English
OAI Identifier:oai:upcommons.upc.edu:2117/101281
Online Access:https://hdl.handle.net/2117/101281
https://dx.doi.org/10.1017/S1351324914000011
Access Level:Open access
Keyword:Natural language processing (Computer science)
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
id ES_205d76b8494d7b2dc2c76c8c48c797fb
oai_identifier_str oai:upcommons.upc.edu:2117/101281
network_acronym_str ES
network_name_str España
repository_id_str
spelling Selection of correction candidates for the normalization of Spanish user generated contentMelero, MaiteRuiz Costa-Jussà, Marta|||0000-0002-5703-520XLambert, PatrikQuixal, MartíNatural language processing (Computer science)Tractament del llenguatge natural (Informàtica)Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge naturalWe present research aiming to build tools for the normalization of User-Generated Content (UGC). We argue that processing this type of text requires the revisiting of the initial steps of Natural Language Processing, since UGC (micro-blog, blog, and, generally, Web 2.0 user-generated texts) presents a number of nonstandard communicative and linguistic characteristics – often closer to oral and colloquial language than to edited text. We present a corpus of UGC text in Spanish from three different sources: Twitter, consumer reviews, and blogs, and describe its main characteristics. We motivate the need for UGC text normalization by analyzing the problems found when processing this type of text through a conventional language processing pipeline, particularly in the tasks of lemmatization and morphosyntactic tagging. Our aim with this paper is to seize the power of already existing spell and grammar correction engines and endow them with automatic normalization capabilities in order to pave the way for the application of standard Natural Language Processing tools to typical UGC text. Particularly, we propose a strategy for automatically normalizing UGC by adding a module on top of a pre-existing spell-checker that selects the most plausible correction from an unranked list of candidates provided by the spell-checker. To build this selector module we train four language models, each one containing a different type of linguistic information in a trade-off with its generalization capabilities. Our experiments show that the models trained on truecase and lowercase word forms are more discriminative than the others at selecting the best candidate. We have also experimented with a parametrized combination of the models by both optimizing directly on the selection task and doing a linear interpolation of the models. The resulting parametrized combinations obtain results close to the best performing model but do not improve on those results, as measured on the test set. The precision of the selector module in ranking number one the expected correction proposal on the test corpora reaches 82.5% for Twitter text (baseline 57%) and 88% for non-Twitter text (baseline 64%).Peer Reviewed20162016-01-0120172017-02-21journal articlehttp://purl.org/coar/resource_type/c_6501AMhttp://purl.org/coar/version/c_ab4af688f83e57aainfo:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/101281https://dx.doi.org/10.1017/S1351324914000011reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/1012812026-05-27T15:37:01Z
dc.title.none.fl_str_mv Selection of correction candidates for the normalization of Spanish user generated content
title Selection of correction candidates for the normalization of Spanish user generated content
spellingShingle Selection of correction candidates for the normalization of Spanish user generated content
Melero, Maite
Natural language processing (Computer science)
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
title_short Selection of correction candidates for the normalization of Spanish user generated content
title_full Selection of correction candidates for the normalization of Spanish user generated content
title_fullStr Selection of correction candidates for the normalization of Spanish user generated content
title_full_unstemmed Selection of correction candidates for the normalization of Spanish user generated content
title_sort Selection of correction candidates for the normalization of Spanish user generated content
dc.creator.none.fl_str_mv Melero, Maite
Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Lambert, Patrik
Quixal, Martí
author Melero, Maite
author_facet Melero, Maite
Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Lambert, Patrik
Quixal, Martí
author_role author
author2 Ruiz Costa-Jussà, Marta|||0000-0002-5703-520X
Lambert, Patrik
Quixal, Martí
author2_role author
author
author
dc.subject.none.fl_str_mv Natural language processing (Computer science)
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
topic Natural language processing (Computer science)
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
description We present research aiming to build tools for the normalization of User-Generated Content (UGC). We argue that processing this type of text requires the revisiting of the initial steps of Natural Language Processing, since UGC (micro-blog, blog, and, generally, Web 2.0 user-generated texts) presents a number of nonstandard communicative and linguistic characteristics – often closer to oral and colloquial language than to edited text. We present a corpus of UGC text in Spanish from three different sources: Twitter, consumer reviews, and blogs, and describe its main characteristics. We motivate the need for UGC text normalization by analyzing the problems found when processing this type of text through a conventional language processing pipeline, particularly in the tasks of lemmatization and morphosyntactic tagging. Our aim with this paper is to seize the power of already existing spell and grammar correction engines and endow them with automatic normalization capabilities in order to pave the way for the application of standard Natural Language Processing tools to typical UGC text. Particularly, we propose a strategy for automatically normalizing UGC by adding a module on top of a pre-existing spell-checker that selects the most plausible correction from an unranked list of candidates provided by the spell-checker. To build this selector module we train four language models, each one containing a different type of linguistic information in a trade-off with its generalization capabilities. Our experiments show that the models trained on truecase and lowercase word forms are more discriminative than the others at selecting the best candidate. We have also experimented with a parametrized combination of the models by both optimizing directly on the selection task and doing a linear interpolation of the models. The resulting parametrized combinations obtain results close to the best performing model but do not improve on those results, as measured on the test set. The precision of the selector module in ranking number one the expected correction proposal on the test corpora reaches 82.5% for Twitter text (baseline 57%) and 88% for non-Twitter text (baseline 64%).
publishDate 2016
dc.date.none.fl_str_mv 2016
2016-01-01
2017
2017-02-21
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
AM
http://purl.org/coar/version/c_ab4af688f83e57aa
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/101281
https://dx.doi.org/10.1017/S1351324914000011
url https://hdl.handle.net/2117/101281
https://dx.doi.org/10.1017/S1351324914000011
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869404454915145728
score 15,301603

Similar Items