Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis

Advances in deep learning have made it simple to generate deepfakes, hyper-realistic videos in which photos and video snippets are processed to create fake videos that look legitimate. Although different deepfake creation methods have been proposed, it is still an open question whether synthesized f...

Descripción completa

Detalles Bibliográficos
Autores: López Gil, Juan Miguel, Gil Iranzo, Rosa María, García González, Roberto
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2024
País:España
Institución:Universitat de Lleida (UdL)
Repositorio:Repositori Obert UdL
OAI Identifier:oai:repositori.udl.cat:10459.1/466916
Acceso en línea:https://doi.org/10.22967/HCIS.2024.14.035
https://hdl.handle.net/10459.1/466916
Access Level:acceso abierto
Palabra clave:Deepfakes
Emotion Recognition
Image Classification
id ES_1300554d6f2b132ecc73abffcef0efa6
oai_identifier_str oai:repositori.udl.cat:10459.1/466916
network_acronym_str ES
network_name_str España
repository_id_str
spelling Analysis of the Reliability of Deepfake Facial Emotion Expression SynthesisLópez Gil, Juan Miguel Gil Iranzo, Rosa MaríaGarcía González, RobertoDeepfakesEmotion RecognitionImage ClassificationAdvances in deep learning have made it simple to generate deepfakes, hyper-realistic videos in which photos and video snippets are processed to create fake videos that look legitimate. Although different deepfake creation methods have been proposed, it is still an open question whether synthesized facial expressions adequately display emotions. To address this issue, we have used one of the most typical facial deepfake creation strategies, an encoder-decoder architecture. Then, we applied this algorithm to create deepfakes from a well-known emotional database in which performers in video recordings display specified emotions. We have tested the facial emotion expression synthesis of the deepfakes comparing the emotions of the photograms in the videos with their original counterparts. Emotion analysis has been performed using two well-known emotion classifications used in the facial emotion recognition area. The results show that facial emotion expression is not successfully synthesized using the face swap algorithm. Our study provides the first empirical evidence of the degree to which different emotions are adequately synthesized in deepfakes in which facial expressions are faked. The presented work displays the limitations of current face swap algorithms to properly synthesize emotions, which have implications in both the synthesis and the detection of deepfake recordings.This work has been partially supported by project "ANGRU: Applying kNowledge Graphs to research data ReUsability" with reference PID2020-117912RB-C22 and funded by MCIN/AEI/ 10.13039/501100011033.KCIA(Korea Computer Industry Association)2024info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionhttps://doi.org/10.22967/HCIS.2024.14.035https://hdl.handle.net/10459.1/466916reponame:Repositori Obert UdL instname:Universitat de Lleida (UdL)Inglésinfo:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2020-117912RB-C22Reproducció del document publicat a https://doi.org/10.22967/HCIS.2024.14.035Human-centric Computing and Information Sciences, 2024, vol. 14, article number 35cc-by-nc (c) KCIA(Korea Computer Industry Association), 2024Attribution-NonCommercial 4.0 Internationalinfo:eu-repo/semantics/openAccesshttp://creativecommons.org/licenses/by-nc/4.0/oai:repositori.udl.cat:10459.1/4669162026-06-24T12:42:17Z
dc.title.none.fl_str_mv Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
title Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
spellingShingle Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
López Gil, Juan Miguel
Deepfakes
Emotion Recognition
Image Classification
title_short Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
title_full Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
title_fullStr Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
title_full_unstemmed Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
title_sort Analysis of the Reliability of Deepfake Facial Emotion Expression Synthesis
dc.creator.none.fl_str_mv López Gil, Juan Miguel
Gil Iranzo, Rosa María
García González, Roberto
author López Gil, Juan Miguel
author_facet López Gil, Juan Miguel
Gil Iranzo, Rosa María
García González, Roberto
author_role author
author2 Gil Iranzo, Rosa María
García González, Roberto
author2_role author
author
dc.subject.none.fl_str_mv Deepfakes
Emotion Recognition
Image Classification
topic Deepfakes
Emotion Recognition
Image Classification
description Advances in deep learning have made it simple to generate deepfakes, hyper-realistic videos in which photos and video snippets are processed to create fake videos that look legitimate. Although different deepfake creation methods have been proposed, it is still an open question whether synthesized facial expressions adequately display emotions. To address this issue, we have used one of the most typical facial deepfake creation strategies, an encoder-decoder architecture. Then, we applied this algorithm to create deepfakes from a well-known emotional database in which performers in video recordings display specified emotions. We have tested the facial emotion expression synthesis of the deepfakes comparing the emotions of the photograms in the videos with their original counterparts. Emotion analysis has been performed using two well-known emotion classifications used in the facial emotion recognition area. The results show that facial emotion expression is not successfully synthesized using the face swap algorithm. Our study provides the first empirical evidence of the degree to which different emotions are adequately synthesized in deepfakes in which facial expressions are faked. The presented work displays the limitations of current face swap algorithms to properly synthesize emotions, which have implications in both the synthesis and the detection of deepfake recordings.
publishDate 2024
dc.date.none.fl_str_mv 2024
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv https://doi.org/10.22967/HCIS.2024.14.035
https://hdl.handle.net/10459.1/466916
url https://doi.org/10.22967/HCIS.2024.14.035
https://hdl.handle.net/10459.1/466916
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2020-117912RB-C22
Reproducció del document publicat a https://doi.org/10.22967/HCIS.2024.14.035
Human-centric Computing and Information Sciences, 2024, vol. 14, article number 35
dc.rights.none.fl_str_mv cc-by-nc (c) KCIA(Korea Computer Industry Association), 2024
Attribution-NonCommercial 4.0 International
info:eu-repo/semantics/openAccess
http://creativecommons.org/licenses/by-nc/4.0/
rights_invalid_str_mv cc-by-nc (c) KCIA(Korea Computer Industry Association), 2024
Attribution-NonCommercial 4.0 International
http://creativecommons.org/licenses/by-nc/4.0/
eu_rights_str_mv openAccess
dc.publisher.none.fl_str_mv KCIA(Korea Computer Industry Association)
publisher.none.fl_str_mv KCIA(Korea Computer Industry Association)
dc.source.none.fl_str_mv reponame:Repositori Obert UdL
instname:Universitat de Lleida (UdL)
instname_str Universitat de Lleida (UdL)
reponame_str Repositori Obert UdL
collection Repositori Obert UdL
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869403650795765760
score 15,811543

Similares en LA Referencia