Fine-tuning CLIP models for peripheral blood cell images retrieval based on morphological descriptions
This master's thesis explores the fine-tuning of Contrastive Language-Image Pre-training (CLIP) models for retrieving peripheral blood cell images based on morphological descriptions, aiming to assist hematologists in diagnostic processes. The study addresses the limitations of manual perip...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2025 |
| País: | España |
| Institución: | Universitat Oberta de Catalunya (UOC) |
| Repositorio: | O2, repositorio institucional de la UOC |
| OAI Identifier: | oai:openaccess.uoc.edu:10609/153609 |
| Acceso en línea: | https://hdl.handle.net/10609/153609 |
| Access Level: | acceso abierto |
| Palabra clave: | vision language models blood cell morphology medical image retrieval Bioinformatics -- TFM Bioinformàtica -- TFM |
| Sumario: | This master's thesis explores the fine-tuning of Contrastive Language-Image Pre-training (CLIP) models for retrieving peripheral blood cell images based on morphological descriptions, aiming to assist hematologists in diagnostic processes. The study addresses the limitations of manual peripheral blood smear analysis, which is time-consuming and prone to variability, by leveraging state-of-the-art AI techniques. The methodology involved preparing a dataset of lymphocyte images with textual descriptions of 12 morphological features. Pre-trained CLIP models (ViT-B/32, ViT-L/14, ViT-B/16) were fine-tuned using Cosine Similarity, Contrastive Loss, and Multiple Negatives Ranking Loss (MNRL). Performance was evaluated using a novel descriptor-specific recall metric. Initial zero-shot performance of the pre-trained CLIP ViT-B/32 was limited, with a Recall@10 of 0.080. Fine-tuning yielded substantial improvements. The MNRL function proved most effective, and the CLIP ViT-B/32 architecture offered a strong balance of performance and efficiency, achieving a Recall@5 increasing from 0.055 (baseline) to 0.356 and a Recall@10 of 0.660. Performance varied across descriptors, with features like 'Cytoplasmic Hairiness' being more reliably retrieved. A web application was developed to demonstrate these capabilities. The study concludes that fine-tuned CLIP models hold significant potential for specialized medical image retrieval. Future work includes exploring advanced data augmentation, refining captioning strategies, and clinical validation. |
|---|