Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques
[EN] Digitisation processes of movable heritage are becoming increasingly popular to document the artworks stored in our museums. An increasing number of strategies for the three-dimensional (3D) acquisition and modelling of these invaluable assets have been developed in the last few years, to effic...
| Autores: | , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2021 |
| País: | España |
| Institución: | Universitat Politècnica de València (UPV) |
| Repositorio: | RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia |
| Idioma: | inglés |
| OAI Identifier: | oai:riunet.upv.es:10251/169360 |
| Acceso en línea: | https://riunet.upv.es/handle/10251/169360 |
| Access Level: | acceso abierto |
| Palabra clave: | Close-range photogrammetry Deep learning Semantic segmentation Automatic masking Movable heritage Cultural heritage documentation Fotogrametría de objeto cercano Aprendizaje profundo Segmentación semántica Enmascaramiento automático Patrimonio mueble Documentación del patrimonio cultural |
| id |
ES_6760a650d54bbc4df2ae6a4f56c77d0f |
|---|---|
| oai_identifier_str |
oai:riunet.upv.es:10251/169360 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| dc.title.none.fl_str_mv |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques Segmentación semántica multiclase en la digitalización del patrimonio mueble utilizando técnicas de aprendizaje profundo |
| title |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques |
| spellingShingle |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques Patrucco, Giacomo Close-range photogrammetry Deep learning Semantic segmentation Automatic masking Movable heritage Cultural heritage documentation Fotogrametría de objeto cercano Aprendizaje profundo Segmentación semántica Enmascaramiento automático Patrimonio mueble Documentación del patrimonio cultural |
| title_short |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques |
| title_full |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques |
| title_fullStr |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques |
| title_full_unstemmed |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques |
| title_sort |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniques |
| dc.creator.none.fl_str_mv |
Patrucco, Giacomo Setragno, Francesco |
| author |
Patrucco, Giacomo |
| author_facet |
Patrucco, Giacomo Setragno, Francesco |
| author_role |
author |
| author2 |
Setragno, Francesco |
| author2_role |
author |
| dc.contributor.none.fl_str_mv |
Repositorio Institucional de la Universitat Politècnica de València Riunet |
| dc.subject.none.fl_str_mv |
Close-range photogrammetry Deep learning Semantic segmentation Automatic masking Movable heritage Cultural heritage documentation Fotogrametría de objeto cercano Aprendizaje profundo Segmentación semántica Enmascaramiento automático Patrimonio mueble Documentación del patrimonio cultural |
| topic |
Close-range photogrammetry Deep learning Semantic segmentation Automatic masking Movable heritage Cultural heritage documentation Fotogrametría de objeto cercano Aprendizaje profundo Segmentación semántica Enmascaramiento automático Patrimonio mueble Documentación del patrimonio cultural |
| description |
[EN] Digitisation processes of movable heritage are becoming increasingly popular to document the artworks stored in our museums. An increasing number of strategies for the three-dimensional (3D) acquisition and modelling of these invaluable assets have been developed in the last few years, to efficiently respond to this documentation need and contribute to deepening the knowledge of the masterpieces investigated constantly by researchers operating in many fieldworks. Nowadays, one of the most effective solutions is represented by the development of image-based techniques, usually connected to a Structure-from-Motion (SfM) photogrammetric approach. However, while the acquisition of the images is relatively rapid, it is the processes connected to the data processing that are very time-consuming and require substantial manual involvement of the operator. The development of deep learning-based strategies can be an effective solution to enhance the level of automatism. In the case of the current research, which has been carried out in the framework of the digitisation of a collection of wooden maquettes stored in the ‘Museo Egizio di Torino’ using a photogrammetric approach, an automatic masking strategy using deep learning techniques is proposed, to increase the level of automatism and therefore, optimise the photogrammetric pipeline. Starting from a manually annotated dataset a neural network has been trained to automatically perform a semantic classification with the aim to isolate the maquettes from the background. The proposed methodology has allowed obtaining automatically segmented masks with a high degree of accuracy. The followed workflow is described (as regards acquisition strategies, dataset processing, and neural network training), and the accuracy of the results is evaluated and discussed. In addition, the possibility of performing a multiclass segmentation on the digital images to recognise different categories of objects in the images and define a semantic hierarchy is proposed to perform automatic classification of different elements in the acquired images. |
| publishDate |
2021 |
| dc.date.none.fl_str_mv |
2021 2021-07-14 |
| dc.type.none.fl_str_mv |
journal article http://purl.org/coar/resource_type/c_6501 VoR http://purl.org/coar/version/c_970fb48d4fbd8a85 |
| dc.type.openaire.fl_str_mv |
info:eu-repo/semantics/article |
| format |
article |
| dc.identifier.none.fl_str_mv |
https://riunet.upv.es/handle/10251/169360 |
| url |
https://riunet.upv.es/handle/10251/169360 |
| dc.language.none.fl_str_mv |
Inglés eng |
| language_invalid_str_mv |
Inglés |
| language |
eng |
| dc.rights.none.fl_str_mv |
open access http://purl.org/coar/access_right/c_abf2 Reconocimiento - No comercial - Sin obra derivada (by-nc-nd) http://creativecommons.org/licenses/by-nc-nd/4.0/ |
| dc.rights.openaire.fl_str_mv |
info:eu-repo/semantics/openAccess |
| rights_invalid_str_mv |
open access http://purl.org/coar/access_right/c_abf2 Reconocimiento - No comercial - Sin obra derivada (by-nc-nd) http://creativecommons.org/licenses/by-nc-nd/4.0/ |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf |
| dc.publisher.none.fl_str_mv |
Universitat Politècnica de València |
| publisher.none.fl_str_mv |
Universitat Politècnica de València |
| dc.source.none.fl_str_mv |
reponame:RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia instname:Universitat Politècnica de València (UPV) |
| instname_str |
Universitat Politècnica de València (UPV) |
| reponame_str |
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia |
| collection |
RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869409885709402112 |
| spelling |
Multiclass semantic segmentation for digitisation of movable heritage using deep learning techniquesSegmentación semántica multiclase en la digitalización del patrimonio mueble utilizando técnicas de aprendizaje profundoPatrucco, GiacomoSetragno, FrancescoClose-range photogrammetryDeep learningSemantic segmentationAutomatic maskingMovable heritageCultural heritage documentationFotogrametría de objeto cercanoAprendizaje profundoSegmentación semánticaEnmascaramiento automáticoPatrimonio muebleDocumentación del patrimonio cultural[EN] Digitisation processes of movable heritage are becoming increasingly popular to document the artworks stored in our museums. An increasing number of strategies for the three-dimensional (3D) acquisition and modelling of these invaluable assets have been developed in the last few years, to efficiently respond to this documentation need and contribute to deepening the knowledge of the masterpieces investigated constantly by researchers operating in many fieldworks. Nowadays, one of the most effective solutions is represented by the development of image-based techniques, usually connected to a Structure-from-Motion (SfM) photogrammetric approach. However, while the acquisition of the images is relatively rapid, it is the processes connected to the data processing that are very time-consuming and require substantial manual involvement of the operator. The development of deep learning-based strategies can be an effective solution to enhance the level of automatism. In the case of the current research, which has been carried out in the framework of the digitisation of a collection of wooden maquettes stored in the ‘Museo Egizio di Torino’ using a photogrammetric approach, an automatic masking strategy using deep learning techniques is proposed, to increase the level of automatism and therefore, optimise the photogrammetric pipeline. Starting from a manually annotated dataset a neural network has been trained to automatically perform a semantic classification with the aim to isolate the maquettes from the background. The proposed methodology has allowed obtaining automatically segmented masks with a high degree of accuracy. The followed workflow is described (as regards acquisition strategies, dataset processing, and neural network training), and the accuracy of the results is evaluated and discussed. In addition, the possibility of performing a multiclass segmentation on the digital images to recognise different categories of objects in the images and define a semantic hierarchy is proposed to perform automatic classification of different elements in the acquired images.[ES] Los procesos de digitalización del patrimonio mueble son cada vez más populares para documentar las obras de arte almacenadas en nuestros museos. En los últimos años se han desarrollado un número creciente de estrategias de adquisición y modelado tridimensional (3D) de estos activos de valor incalculable, que responden de manera eficiente a esta necesidad de documentación y contribuyen a profundizar en el conocimiento de las obras maestras investigadas constantemente por investigadores que operan en muchos trabajos de campo. Hoy en día, una de las soluciones más efectivas está relacionada con el desarrollo de técnicas basadas en imágenes, generalmente conectadas a un enfoque fotogramétrico de estructura-y-movimiento (SfM). Sin embargo, si bien la adquisición de las imágenes es relativamente rápida, son los procesos relacionados con el procesamiento de los datos los que consumen mucho tiempo y requieren una participación manual sustancial del operador. El desarrollo de estrategias basadas en el aprendizaje profundo puede ser una solución eficaz para mejorar el nivel de automatismo. En el caso de la presente investigación, que se ha llevado a cabo en el marco de la digitalización de una colección de maquetas de madera almacenadas en el 'Museo Egizio di Torino' mediante un enfoque fotogramétrico, se propone una estrategia de enmascaramiento automático mediante técnicas de aprendizaje profundo, que incrementa el nivel de automatismo y por tanto optimiza el flujo fotogramétrico. A partir de un conjunto de datos anotados manualmente, se ha entrenado una red neuronal que realiza automáticamente una clasificación semántica con el objetivo de aislar las maquetas del fondo. La metodología propuesta ha permitido obtener más caras segmentadas automáticamente con alto grado de precisión. Se describe el flujo de trabajo seguido (en cuanto a estrategias de toma, procesamiento del conjuntos de datos y entrenamiento de las redes neuronales), y se evalúa y discute la precisión de los resultados. Además, se propone la posibilidad de realizar una segmentación multiclase sobre las imágenes digitales que permitan reconocer diferentes categorías de objetos en las imágenes y definir una jerarquía semántica que clasifique automáticamente diferentes elementos en la toma de las imágenes.The authors thank Volta® A.I. (and in particular Silvio Revelli) for the contribution to this work and for providing high-end hardware for neural network training. In addition, they would like to thank Alessia Fassone of Museo Egizio di Torino and all the people involved in the B.A.C.K. TO T.H.E. F.U.T.U.RE. project (in particular, Fulvio Rinaudo, who coordinated the Geomatic team). Finally, they wish to express their gratitude to Nannina Spanò and Filiberto Chiabrando for the helpful confrontation during the presented research.Universitat Politècnica de ValènciaRepositorio Institucional de la Universitat Politècnica de València Riunet20212021-07-14journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://riunet.upv.es/handle/10251/169360reponame:RiuNet. Repositorio Institucional de la Universitat Politécnica de Valénciainstname:Universitat Politècnica de València (UPV)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2Reconocimiento - No comercial - Sin obra derivada (by-nc-nd) http://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:riunet.upv.es:10251/1693602026-06-13T07:49:27Z |
| score |
15.301603 |