Towards Facial Emotion Recognition using a Transformer-based Model with 3D Facial Pointclouds
Facial Emotion Recognition (FER) stands as a pivotal area of research within Affective Computing (AAC) due to its broad applicability in domains such as human-robot interaction (HRI), mental health assessment, and fatigue monitoring. Current state-of-the-art FER approaches are based on deep learning...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2024 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/422399 |
| Acceso en línea: | https://hdl.handle.net/2117/422399 |
| Access Level: | acceso abierto |
| Palabra clave: | Human face recognition (Computer science) Transfer learning (Machine learning) Deep learning (Machine learning) 3D Facial Emotion Recognition Joint Communication and Sensing (JC&S) Transformers Deep Learning Transfer Learning Reconeixement facial (Informàtica) Aprenentatge automàtic Aprenentatge profund Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Reconeixement de formes |
| Sumario: | Facial Emotion Recognition (FER) stands as a pivotal area of research within Affective Computing (AAC) due to its broad applicability in domains such as human-robot interaction (HRI), mental health assessment, and fatigue monitoring. Current state-of-the-art FER approaches are based on deep learning methods and primarily leverage 2D image data. However, the acquisition of such data through off-person sensors poses significant privacy challenges under EU regulations. In response, we envision a trend shift towards on-person sensors integrated into wearables, facilitated by the Joint Communication and Sensing (JC&S) paradigm. This paradigm proposes the use of high-frequency electromagnetic waves for fine-grained structural 3D imaging of the human face, enabling continuous and privacy-preserving emotion tracking. To support this emerging paradigm, this work addresses the two open challenges facing 3D-based FER: the limited size of 3D databases labelled for FER, and the need for understanding the use cases where Transfer Learning for 3D-based DL models is applicable. To overcome the first challenge, we showcase the potential of using a previously developed in-house method for generating 3D FER databases from existing 2D image FER databases. Obtained through this method, we use a 3D version of the popular 2D FER AffectNet database for 3D FER DL model training, achieving a 31.62% accuracy. Furthermore we introduce a data refining pipeline to facilitate the model's learning process. By isolating the facial region of the 3D point clouds, this refining process allows further boosting classification accuracy by 12%, which confirms the potential of our approach. To overcome the second challenge, we pioneer the use of transfer learning from non-FER 3D domains to FER-3D domains. We fine-tune the transformer-based DL model I2P-MAE, pre-trained on the ShapeNet database of CAD objects, and achieve the promising result of 90.12% accuracy on the well-established 3D FER BU-3DFE dataset. |
|---|