Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles

Unmanned Aerial Vehicles (UAVs) are able to provide instantaneous visual cues and a high-level data throughput that could be further leveraged to address complex tasks, such as semantically rich scene understanding. In this work, we built on the use of Large Language Models (LLMs) and Visual Languag...

Descripción completa

Detalles Bibliográficos
Autor: de Curtò y Díaz, J.
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2023
País:España
Institución:Universitat Oberta de Catalunya (UOC)
Repositorio:O2, repositorio institucional de la UOC
OAI Identifier:oai:openaccess.uoc.edu:10609/151394
Acceso en línea:http://hdl.handle.net/10609/151394
https://doi.org/10.3390/drones7020114
Access Level:acceso abierto
Palabra clave:scene understanding
large language models
visual language models
CLIP
GPT-3
YOLOv7
UAV
id ES_aa362f2ea2acefc1ecff0e190b334202
oai_identifier_str oai:openaccess.uoc.edu:10609/151394
network_acronym_str ES
network_name_str España
repository_id_str
spelling Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehiclesde Curtò y Díaz, J.scene understandinglarge language modelsvisual language modelsCLIPGPT-3YOLOv7UAVUnmanned Aerial Vehicles (UAVs) are able to provide instantaneous visual cues and a high-level data throughput that could be further leveraged to address complex tasks, such as semantically rich scene understanding. In this work, we built on the use of Large Language Models (LLMs) and Visual Language Models (VLMs), together with a state-of-the-art detection pipeline, to provide thorough zero-shot UAV scene literary text descriptions. The generated texts achieve a GUNNING Fog median grade level in the range of 7–12. Applications of this framework could be found in the filming industry and could enhance user experience in theme parks or in the advertisement sector. We demonstrate a low-cost highly efficient state-of-the-art practical implementation of microdrones in a well-controlled and challenging setting, in addition to proposing the use of standardized readability metrics to assess LLM-enhanced descriptions.MDPI AGde Zarzà i Cubero, I.Calafate, Carlos202420242023info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfapplication/pdfhttp://hdl.handle.net/10609/151394https://doi.org/10.3390/drones7020114reponame:O2, repositorio institucional de la UOCinstname:Universitat Oberta de Catalunya (UOC)InglésDrones 7, no. 2https://www.mdpi.com/2504-446X/7/2/114http://creativecommons.org/licenses/by-sa/3.0/es/info:eu-repo/semantics/openAccessoai:openaccess.uoc.edu:10609/1513942026-05-28T12:42:01Z
dc.title.none.fl_str_mv Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
title Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
spellingShingle Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
de Curtò y Díaz, J.
scene understanding
large language models
visual language models
CLIP
GPT-3
YOLOv7
UAV
title_short Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
title_full Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
title_fullStr Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
title_full_unstemmed Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
title_sort Semantic Scene Understanding with Large Language Models on Unmanned Aerial Vehicles
dc.creator.none.fl_str_mv de Curtò y Díaz, J.
author de Curtò y Díaz, J.
author_facet de Curtò y Díaz, J.
author_role author
dc.contributor.none.fl_str_mv de Zarzà i Cubero, I.
Calafate, Carlos
dc.subject.none.fl_str_mv scene understanding
large language models
visual language models
CLIP
GPT-3
YOLOv7
UAV
topic scene understanding
large language models
visual language models
CLIP
GPT-3
YOLOv7
UAV
description Unmanned Aerial Vehicles (UAVs) are able to provide instantaneous visual cues and a high-level data throughput that could be further leveraged to address complex tasks, such as semantically rich scene understanding. In this work, we built on the use of Large Language Models (LLMs) and Visual Language Models (VLMs), together with a state-of-the-art detection pipeline, to provide thorough zero-shot UAV scene literary text descriptions. The generated texts achieve a GUNNING Fog median grade level in the range of 7–12. Applications of this framework could be found in the filming industry and could enhance user experience in theme parks or in the advertisement sector. We demonstrate a low-cost highly efficient state-of-the-art practical implementation of microdrones in a well-controlled and challenging setting, in addition to proposing the use of standardized readability metrics to assess LLM-enhanced descriptions.
publishDate 2023
dc.date.none.fl_str_mv 2023
2024
2024
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/10609/151394
https://doi.org/10.3390/drones7020114
url http://hdl.handle.net/10609/151394
https://doi.org/10.3390/drones7020114
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Drones 7, no. 2
https://www.mdpi.com/2504-446X/7/2/114
dc.rights.none.fl_str_mv http://creativecommons.org/licenses/by-sa/3.0/es/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv http://creativecommons.org/licenses/by-sa/3.0/es/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv MDPI AG
publisher.none.fl_str_mv MDPI AG
dc.source.none.fl_str_mv reponame:O2, repositorio institucional de la UOC
instname:Universitat Oberta de Catalunya (UOC)
instname_str Universitat Oberta de Catalunya (UOC)
reponame_str O2, repositorio institucional de la UOC
collection O2, repositorio institucional de la UOC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869416143450537984
score 15,811543

Similares en LA Referencia