Appled NLP and ML for the detection of inappropiarte text in a communications platform

With the expansion of the communication platforms, individuals are very used to exchange information and communicate with other people through these platforms, both for social and business purposes. It’s a known problem, that many people use the anonymity provided by these platforms to use inappropr...

Full description

Bibliographic Details
Author: Urrutia Zubikarai, Aitor
Format: master thesis
Publication Date:2020
Country:España
Institution:Universitat Politècnica de Catalunya (UPC)
Repository:UPCommons. Portal del coneixement obert de la UPC
Language:English
OAI Identifier:oai:upcommons.upc.edu:2117/336082
Online Access:https://hdl.handle.net/2117/336082
Access Level:Open access
Keyword:Machine learning
Natural language processing (Computer science)
Natural Language Processing
Deep Learning
Word Embedding
Model Ensemble
Aprenentatge automàtic
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica
id ES_1d880c361b8417cbe28fa9ed6a3000af
oai_identifier_str oai:upcommons.upc.edu:2117/336082
network_acronym_str ES
network_name_str España
repository_id_str
spelling Appled NLP and ML for the detection of inappropiarte text in a communications platformUrrutia Zubikarai, AitorMachine learningNatural language processing (Computer science)Natural Language ProcessingMachine learningDeep LearningWord EmbeddingModel EnsembleAprenentatge automàticTractament del llenguatge natural (Informàtica)Àrees temàtiques de la UPC::InformàticaWith the expansion of the communication platforms, individuals are very used to exchange information and communicate with other people through these platforms, both for social and business purposes. It’s a known problem, that many people use the anonymity provided by these platforms to use inappropriate and offensive language. The company NextreT S.L. has built a communication platform directed to business use. As the company wants to avoid offensive language in this platform, natural language processing tools in big data environment are going to be used to analyze each written text to detect and remove if required this inappropriate and offensive language. During this project, a variety of techniques used in the State of the Art are analyzed, compared and then tested using a completely new data set in Spanish language created using Tellfy App and Twitter corpus. Initially, different word encoding methods are tested, including word embedding like Word2Vec and FastText. In addition, different hyper parameter configurations are checked as well as model performances with different data sizes. Finally, after a forward feature selection phase, model ensemble techniques are tested. During these tests, it has been shown that the combination of the features that are used is very important to increase the performance of the models. Also, the different word representation techniques are very related to the performance of the models. Furthermore, the sizes of the training sets that are used need to be as representative and as large as possible. Finally, after using different complex Deep Neural Network models, more traditional Logistic Regression models can offer a better performance.Universitat Politècnica de CatalunyaAlquezar Mancho, RenéRiera Molina, Jordi20202020-01-3120212021-01-27master thesishttp://purl.org/coar/resource_type/c_bdccNAhttp://purl.org/coar/version/c_be7fb7dd8ff6fe43info:eu-repo/semantics/masterThesisapplication/pdfhttps://hdl.handle.net/2117/336082reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/3360822026-05-27T15:37:01Z
dc.title.none.fl_str_mv Appled NLP and ML for the detection of inappropiarte text in a communications platform
title Appled NLP and ML for the detection of inappropiarte text in a communications platform
spellingShingle Appled NLP and ML for the detection of inappropiarte text in a communications platform
Urrutia Zubikarai, Aitor
Machine learning
Natural language processing (Computer science)
Natural Language Processing
Machine learning
Deep Learning
Word Embedding
Model Ensemble
Aprenentatge automàtic
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica
title_short Appled NLP and ML for the detection of inappropiarte text in a communications platform
title_full Appled NLP and ML for the detection of inappropiarte text in a communications platform
title_fullStr Appled NLP and ML for the detection of inappropiarte text in a communications platform
title_full_unstemmed Appled NLP and ML for the detection of inappropiarte text in a communications platform
title_sort Appled NLP and ML for the detection of inappropiarte text in a communications platform
dc.creator.none.fl_str_mv Urrutia Zubikarai, Aitor
author Urrutia Zubikarai, Aitor
author_facet Urrutia Zubikarai, Aitor
author_role author
dc.contributor.none.fl_str_mv Alquezar Mancho, René
Riera Molina, Jordi
dc.subject.none.fl_str_mv Machine learning
Natural language processing (Computer science)
Natural Language Processing
Machine learning
Deep Learning
Word Embedding
Model Ensemble
Aprenentatge automàtic
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica
topic Machine learning
Natural language processing (Computer science)
Natural Language Processing
Machine learning
Deep Learning
Word Embedding
Model Ensemble
Aprenentatge automàtic
Tractament del llenguatge natural (Informàtica)
Àrees temàtiques de la UPC::Informàtica
description With the expansion of the communication platforms, individuals are very used to exchange information and communicate with other people through these platforms, both for social and business purposes. It’s a known problem, that many people use the anonymity provided by these platforms to use inappropriate and offensive language. The company NextreT S.L. has built a communication platform directed to business use. As the company wants to avoid offensive language in this platform, natural language processing tools in big data environment are going to be used to analyze each written text to detect and remove if required this inappropriate and offensive language. During this project, a variety of techniques used in the State of the Art are analyzed, compared and then tested using a completely new data set in Spanish language created using Tellfy App and Twitter corpus. Initially, different word encoding methods are tested, including word embedding like Word2Vec and FastText. In addition, different hyper parameter configurations are checked as well as model performances with different data sizes. Finally, after a forward feature selection phase, model ensemble techniques are tested. During these tests, it has been shown that the combination of the features that are used is very important to increase the performance of the models. Also, the different word representation techniques are very related to the performance of the models. Furthermore, the sizes of the training sets that are used need to be as representative and as large as possible. Finally, after using different complex Deep Neural Network models, more traditional Logistic Regression models can offer a better performance.
publishDate 2020
dc.date.none.fl_str_mv 2020
2020-01-31
2021
2021-01-27
dc.type.none.fl_str_mv master thesis
http://purl.org/coar/resource_type/c_bdcc
NA
http://purl.org/coar/version/c_be7fb7dd8ff6fe43
dc.type.openaire.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/336082
url https://hdl.handle.net/2117/336082
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Universitat Politècnica de Catalunya
publisher.none.fl_str_mv Universitat Politècnica de Catalunya
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869404284986064896
score 15,300719

Similar Items