VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings

This research explores the integration of CLIP, a pretrained model, into video content analysis. In a landscape inundated with multimedia data, pinpointing specific moments within videos is a persistent challenge. By leveraging CLIP's semantic and visual search capabilities, this study endeavor...

Descripción completa

Detalles Bibliográficos
Autor: Caravaca Müller, Oriol
Tipo de recurso: tesis de maestría
Fecha de publicación:2024
País:España
Institución:Universitat Oberta de Catalunya (UOC)
Repositorio:O2, repositorio institucional de la UOC
OAI Identifier:oai:openaccess.uoc.edu:10609/149808
Acceso en línea:http://hdl.handle.net/10609/149808
Access Level:acceso abierto
Palabra clave:video analysis
moment retrieval
CLIP
Computer vision -- TFM
Visió per ordinador -- TFM
id ES_7f3f0dfceefe7daec6162019fb5557ea
oai_identifier_str oai:openaccess.uoc.edu:10609/149808
network_acronym_str ES
network_name_str España
repository_id_str
spelling VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordingsCaravaca Müller, Oriolvideo analysismoment retrievalCLIPComputer vision -- TFMVisió per ordinador -- TFMThis research explores the integration of CLIP, a pretrained model, into video content analysis. In a landscape inundated with multimedia data, pinpointing specific moments within videos is a persistent challenge. By leveraging CLIP's semantic and visual search capabilities, this study endeavors to refine content retrieval methods. Emphasizing efficiency and applicability, this study aims to make this process more precise and practical. With this research we also reviewed the state-of-the-art methods and produced empirical analysis on the effects of postprocessing on the similarity vectors obtained from CLIP encoders. Finally, we developed two distinct methods aimed at moment retrieval tasks in audiovisual data, obtaining a model that is able to outperform previous works in Zero-shot moment revival, reaching 57.3 at R@1 IoU=0.5 and 51.6 at mAP@0.5.Aquesta investigació explora la integració de CLIP, un model preentrenat, en l'anàlisi de contingut de vídeo. En un paisatge inundat de dades multimèdia, identificar moments específics dels vídeos és un repte persistent. Aprofitant les capacitats de cerca semàntica i visual de CLIP, aquest estudi intenta perfeccionar els mètodes de recuperació de contingut. Subratllant l'eficiència i l'aplicabilitat, fent aquest procés més precís i pràctic. En aquesta investigació també s’ha revisat l’estat de l’art i s’ha produit un anàlisis empíric sobre els efectes del postprocessament sobre els vectors de semblança obtinguts a partir dels codificadors de CLIP. Finalment s’han desenvolupat dos mètodes diferents dirigits a tasques de recuperació de moments en dades audiovisuals, obtenint un model que és capaç de superar els treballs anteriors en Zero-shot moment revival, arribant a 57,3 a R@1 IoU=0,5 i 51,6 a mAP@0,5.Universitat Oberta de Catalunya (UOC)Benito Altamirano, Ismael202420242024info:eu-repo/semantics/masterThesisapplication/pdfapplication/pdfhttp://hdl.handle.net/10609/149808reponame:O2, repositorio institucional de la UOCinstname:Universitat Oberta de Catalunya (UOC)InglésCC BY-NC-NDhttp://creativecommons.org/licenses/by-nc-nd/3.0/es/info:eu-repo/semantics/openAccessoai:openaccess.uoc.edu:10609/1498082026-05-28T12:42:01Z
dc.title.none.fl_str_mv VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
title VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
spellingShingle VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
Caravaca Müller, Oriol
video analysis
moment retrieval
CLIP
Computer vision -- TFM
Visió per ordinador -- TFM
title_short VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
title_full VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
title_fullStr VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
title_full_unstemmed VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
title_sort VClipper: Exploiting CLIP Zero-shot capabilities for moment retrieval in video recordings
dc.creator.none.fl_str_mv Caravaca Müller, Oriol
author Caravaca Müller, Oriol
author_facet Caravaca Müller, Oriol
author_role author
dc.contributor.none.fl_str_mv Benito Altamirano, Ismael
dc.subject.none.fl_str_mv video analysis
moment retrieval
CLIP
Computer vision -- TFM
Visió per ordinador -- TFM
topic video analysis
moment retrieval
CLIP
Computer vision -- TFM
Visió per ordinador -- TFM
description This research explores the integration of CLIP, a pretrained model, into video content analysis. In a landscape inundated with multimedia data, pinpointing specific moments within videos is a persistent challenge. By leveraging CLIP's semantic and visual search capabilities, this study endeavors to refine content retrieval methods. Emphasizing efficiency and applicability, this study aims to make this process more precise and practical. With this research we also reviewed the state-of-the-art methods and produced empirical analysis on the effects of postprocessing on the similarity vectors obtained from CLIP encoders. Finally, we developed two distinct methods aimed at moment retrieval tasks in audiovisual data, obtaining a model that is able to outperform previous works in Zero-shot moment revival, reaching 57.3 at R@1 IoU=0.5 and 51.6 at mAP@0.5.
publishDate 2024
dc.date.none.fl_str_mv 2024
2024
2024
dc.type.none.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
dc.identifier.none.fl_str_mv http://hdl.handle.net/10609/149808
url http://hdl.handle.net/10609/149808
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.rights.none.fl_str_mv CC BY-NC-ND
http://creativecommons.org/licenses/by-nc-nd/3.0/es/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv CC BY-NC-ND
http://creativecommons.org/licenses/by-nc-nd/3.0/es/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Universitat Oberta de Catalunya (UOC)
publisher.none.fl_str_mv Universitat Oberta de Catalunya (UOC)
dc.source.none.fl_str_mv reponame:O2, repositorio institucional de la UOC
instname:Universitat Oberta de Catalunya (UOC)
instname_str Universitat Oberta de Catalunya (UOC)
reponame_str O2, repositorio institucional de la UOC
collection O2, repositorio institucional de la UOC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869411809918713856
score 15,301603

Similares en LA Referencia