Applying deep learning techniques to estimate patterns of musical gesture
Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin...
| Autores: | , , |
|---|---|
| Tipo de recurso: | artículo |
| Estado: | Versión publicada |
| Fecha de publicación: | 2021 |
| País: | España |
| Institución: | Universitat Pompeu Fabra |
| Repositorio: | Repositorio Digital de la UPF |
| OAI Identifier: | oai:repositori.upf.edu:10230/46473 |
| Acceso en línea: | http://hdl.handle.net/10230/46473 http://dx.doi.org/10.3389/fpsyg.2020.575971 |
| Access Level: | acceso abierto |
| Palabra clave: | Gesture recognition Bow-strokes Music interaction CNN LSTM Music education ConvLSTM CNN_LSTM |
| id |
ES_2d044245ea11ff2239956ebd5ce213a8 |
|---|---|
| oai_identifier_str |
oai:repositori.upf.edu:10230/46473 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| spelling |
Applying deep learning techniques to estimate patterns of musical gestureDalmazzo, DavidWaddell, GeorgeRamírez, Rafael, 1966-Gesture recognitionBow-strokesMusic interactionCNNLSTMMusic educationConvLSTMCNN_LSTMRepetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance.This work has been partly sponsored by the Spanish TIN project TIMUL (TIN 2013-48152-C2-2-R), the European Union Horizon 2020 research and innovation program under grant agreement No. 688269 (TELMI project), and the Spanish Ministry of Economy and Competitiveness under the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502).Frontiers202120212021info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfapplication/pdfhttp://hdl.handle.net/10230/46473http://dx.doi.org/10.3389/fpsyg.2020.575971reponame:Repositorio Digital de la UPFinstname:Universitat Pompeu FabraInglésFrontiers in Psychology. 2021 Jan 5;11:575971.info:eu-repo/grantAgreement/ES/1PE/TIN2013-48152-C2-2-Rinfo:eu-repo/grantAgreement/EC/H2020/688269© 2021 Dalmazzo, Waddell and Ramírez. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) (https://creativecommons.org/licenses/by/4.0/). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.http://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:repositori.upf.edu:10230/464732026-06-12T07:21:37Z |
| dc.title.none.fl_str_mv |
Applying deep learning techniques to estimate patterns of musical gesture |
| title |
Applying deep learning techniques to estimate patterns of musical gesture |
| spellingShingle |
Applying deep learning techniques to estimate patterns of musical gesture Dalmazzo, David Gesture recognition Bow-strokes Music interaction CNN LSTM Music education ConvLSTM CNN_LSTM |
| title_short |
Applying deep learning techniques to estimate patterns of musical gesture |
| title_full |
Applying deep learning techniques to estimate patterns of musical gesture |
| title_fullStr |
Applying deep learning techniques to estimate patterns of musical gesture |
| title_full_unstemmed |
Applying deep learning techniques to estimate patterns of musical gesture |
| title_sort |
Applying deep learning techniques to estimate patterns of musical gesture |
| dc.creator.none.fl_str_mv |
Dalmazzo, David Waddell, George Ramírez, Rafael, 1966- |
| author |
Dalmazzo, David |
| author_facet |
Dalmazzo, David Waddell, George Ramírez, Rafael, 1966- |
| author_role |
author |
| author2 |
Waddell, George Ramírez, Rafael, 1966- |
| author2_role |
author author |
| dc.subject.none.fl_str_mv |
Gesture recognition Bow-strokes Music interaction CNN LSTM Music education ConvLSTM CNN_LSTM |
| topic |
Gesture recognition Bow-strokes Music interaction CNN LSTM Music education ConvLSTM CNN_LSTM |
| description |
Repetitive practice is one of the most important factors in improving the performance of motor skills. This paper focuses on the analysis and classification of forearm gestures in the context of violin playing. We recorded five experts and three students performing eight traditional classical violin bow-strokes: martelé, staccato, detaché, ricochet, legato, trémolo, collé, and col legno. To record inertial motion information, we utilized the Myo sensor, which reports a multidimensional time-series signal. We synchronized inertial motion recordings with audio data to extract the spatiotemporal dynamics of each gesture. Applying state-of-the-art deep neural networks, we implemented and compared different architectures where convolutional neural networks (CNN) models demonstrated recognition rates of 97.147%, 3DMultiHeaded_CNN models showed rates of 98.553%, and rates of 99.234% were demonstrated by CNN_LSTM models. The collected data (quaternion of the bowing arm of a violinist) contained sufficient information to distinguish the bowing techniques studied, and deep learning methods were capable of learning the movement patterns that distinguish these techniques. Each of the learning algorithms investigated (CNN, 3DMultiHeaded_CNN, and CNN_LSTM) produced high classification accuracies which supported the feasibility of training classifiers. The resulting classifiers may provide the foundation of a digital assistant to enhance musicians' time spent practicing alone, providing real-time feedback on the accuracy and consistency of their musical gestures in performance. |
| publishDate |
2021 |
| dc.date.none.fl_str_mv |
2021 2021 2021 |
| dc.type.none.fl_str_mv |
info:eu-repo/semantics/article info:eu-repo/semantics/publishedVersion |
| format |
article |
| status_str |
publishedVersion |
| dc.identifier.none.fl_str_mv |
http://hdl.handle.net/10230/46473 http://dx.doi.org/10.3389/fpsyg.2020.575971 |
| url |
http://hdl.handle.net/10230/46473 http://dx.doi.org/10.3389/fpsyg.2020.575971 |
| dc.language.none.fl_str_mv |
Inglés |
| language_invalid_str_mv |
Inglés |
| dc.relation.none.fl_str_mv |
Frontiers in Psychology. 2021 Jan 5;11:575971. info:eu-repo/grantAgreement/ES/1PE/TIN2013-48152-C2-2-R info:eu-repo/grantAgreement/EC/H2020/688269 |
| dc.rights.none.fl_str_mv |
http://creativecommons.org/licenses/by/4.0/ info:eu-repo/semantics/openAccess |
| rights_invalid_str_mv |
http://creativecommons.org/licenses/by/4.0/ |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf application/pdf |
| dc.publisher.none.fl_str_mv |
Frontiers |
| publisher.none.fl_str_mv |
Frontiers |
| dc.source.none.fl_str_mv |
reponame:Repositorio Digital de la UPF instname:Universitat Pompeu Fabra |
| instname_str |
Universitat Pompeu Fabra |
| reponame_str |
Repositorio Digital de la UPF |
| collection |
Repositorio Digital de la UPF |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869405281819033600 |
| score |
15,812429 |