Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challeng...
| Autores: | , , , , , , , , |
|---|---|
| Tipo de recurso: | artículo |
| Estado: | Versión publicada |
| Fecha de publicación: | 2023 |
| País: | España |
| Institución: | Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya) |
| Repositorio: | Recercat. Dipósit de la Recerca de Catalunya |
| OAI Identifier: | oai:recercat.cat:10459.1/463463 |
| Acceso en línea: | https://doi.org/10.1016/j.compag.2023.107854 https://hdl.handle.net/10459.1/463463 |
| Access Level: | acceso abierto |
| Palabra clave: | Fruit detection Fruit measurement Yield estimation Fruit visibility Deep learning Precision agriculture |
| id |
ES_601ecedf4a1dbb81b05b689d35c8a9e8 |
|---|---|
| oai_identifier_str |
oai:recercat.cat:10459.1/463463 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| dc.title.none.fl_str_mv |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| title |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| spellingShingle |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation Gené Mola, Jordi Fruit detection Fruit measurement Yield estimation Fruit visibility Deep learning Precision agriculture |
| title_short |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| title_full |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| title_fullStr |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| title_full_unstemmed |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| title_sort |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation |
| dc.creator.none.fl_str_mv |
Gené Mola, Jordi Ferrer Ferrer, Mar Gregorio López, Eduard Blok, Pieter Hemming, Jochen Morros Rubió, Josep Ramon Rosell Polo, Joan Ramon Vilaplana Besler, Verónica Ruiz Hidalgo, Javier |
| author |
Gené Mola, Jordi |
| author_facet |
Gené Mola, Jordi Ferrer Ferrer, Mar Gregorio López, Eduard Blok, Pieter Hemming, Jochen Morros Rubió, Josep Ramon Rosell Polo, Joan Ramon Vilaplana Besler, Verónica Ruiz Hidalgo, Javier |
| author_role |
author |
| author2 |
Ferrer Ferrer, Mar Gregorio López, Eduard Blok, Pieter Hemming, Jochen Morros Rubió, Josep Ramon Rosell Polo, Joan Ramon Vilaplana Besler, Verónica Ruiz Hidalgo, Javier |
| author2_role |
author author author author author author author author |
| dc.subject.none.fl_str_mv |
Fruit detection Fruit measurement Yield estimation Fruit visibility Deep learning Precision agriculture |
| topic |
Fruit detection Fruit measurement Yield estimation Fruit visibility Deep learning Precision agriculture |
| description |
The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for sim ultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing. |
| publishDate |
2023 |
| dc.date.none.fl_str_mv |
2023 |
| dc.type.none.fl_str_mv |
info:eu-repo/semantics/article info:eu-repo/semantics/publishedVersion |
| format |
article |
| status_str |
publishedVersion |
| dc.identifier.none.fl_str_mv |
https://doi.org/10.1016/j.compag.2023.107854 https://hdl.handle.net/10459.1/463463 |
| url |
https://doi.org/10.1016/j.compag.2023.107854 https://hdl.handle.net/10459.1/463463 |
| dc.language.none.fl_str_mv |
Inglés |
| language_invalid_str_mv |
Inglés |
| dc.relation.none.fl_str_mv |
info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/RTI2018-094222-B-I00 info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2020-117142GB-I00 info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica, Técnica y de Innovación 2021-2023/PID2021-126648OB-I00 Reproducció del document publicat a https://doi.org/10.1016/j.compag.2023.107854 Computers and Electronics in Agriculture, 2023, vol. 209, 107854 https://repositori.udl.cat/handle/10459.1/464013 |
| dc.rights.none.fl_str_mv |
cc-by (c) Jordi Gené Mola et al., 2023 Attribution 4.0 International info:eu-repo/semantics/openAccess http://creativecommons.org/licenses/by/4.0/ |
| rights_invalid_str_mv |
cc-by (c) Jordi Gené Mola et al., 2023 Attribution 4.0 International http://creativecommons.org/licenses/by/4.0/ |
| eu_rights_str_mv |
openAccess |
| dc.publisher.none.fl_str_mv |
Elsevier |
| publisher.none.fl_str_mv |
Elsevier |
| dc.source.none.fl_str_mv |
reponame:Recercat. Dipósit de la Recerca de Catalunya instname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya) |
| instname_str |
Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya) |
| reponame_str |
Recercat. Dipósit de la Recerca de Catalunya |
| collection |
Recercat. Dipósit de la Recerca de Catalunya |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869409270033809408 |
| spelling |
Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimationGené Mola, JordiFerrer Ferrer, MarGregorio López, EduardBlok, PieterHemming, JochenMorros Rubió, Josep RamonRosell Polo, Joan RamonVilaplana Besler, VerónicaRuiz Hidalgo, JavierFruit detectionFruit measurementYield estimationFruit visibilityDeep learningPrecision agricultureThe detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for sim ultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_Sizing.This work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gen´e Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study.Elsevier2023info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionhttps://doi.org/10.1016/j.compag.2023.107854https://hdl.handle.net/10459.1/463463reponame:Recercat. Dipósit de la Recerca de Catalunyainstname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)Inglésinfo:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/RTI2018-094222-B-I00info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/PID2020-117142GB-I00info:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica, Técnica y de Innovación 2021-2023/PID2021-126648OB-I00Reproducció del document publicat a https://doi.org/10.1016/j.compag.2023.107854Computers and Electronics in Agriculture, 2023, vol. 209, 107854https://repositori.udl.cat/handle/10459.1/464013cc-by (c) Jordi Gené Mola et al., 2023Attribution 4.0 Internationalinfo:eu-repo/semantics/openAccesshttp://creativecommons.org/licenses/by/4.0/oai:recercat.cat:10459.1/4634632026-05-29T05:05:01Z |
| score |
15.81155 |