Answering questions about images that require outside knowledge
[EN] The emergence of Transformer architectures, pretrained models and multimodal data problems have generated new challenges to solve. One of the most popular in recent years is the visual-linguistic task Visual Question Answering (VQA). Several variants of this task have emerged, one of them being...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2023 |
| País: | España |
| Institución: | Universidad del País Vasco |
| Repositorio: | Addi. Archivo Digital para la Docencia y la Investigación |
| OAI Identifier: | oai:addi.ehu.eus:10810/68845 |
| Acceso en línea: | http://hdl.handle.net/10810/68845 |
| Access Level: | acceso abierto |
| Palabra clave: | transformers multimodal OK-VQA CBM multilabel leverage technique object detection reranking system |
| Sumario: | [EN] The emergence of Transformer architectures, pretrained models and multimodal data problems have generated new challenges to solve. One of the most popular in recent years is the visual-linguistic task Visual Question Answering (VQA). Several variants of this task have emerged, one of them being the Outside Knowledge Visual Question Answering (OK-VQA) task, on which our research will focus. This task adds the complexity that the answer to the question does not appear explicitly in the image, and an external source of knowledge is needed to answer the question. Once the different proposals have been analyzed, the Caption Based Model (CBM) that will serve as the basis for the development is presented. After the problem has been introduced, the proposals are presented, divided into two groups. On the one hand, a multilabel leveraging technique that can be used in multilabel tasks that have optimal and suboptimal solutions, improving model learning. This technique introduces the concept of balance between exploration and exploitation by means of a frequency distribution based on the proportion in which the solutions appear in the ground truth. On the other hand, different image verbalization approaches are analyzed and compared. First, using an object detector, the objects and attributes that appear in an image are obtained. Thus, in addition to providing the CBM model with the image caption (where general image information is represented), we also provide object and attribute information (representing image details). In this way, the balance between general and detailed information is improved. Secondly, due to memory limitations, several reranking systems based on Sentence Similarity and Object bounding box area are presented. These systems seek to improve the quality of the information we pass to the model with respect to the question. After several experiments, we conclude that the new multilabel leverage technique improves model learning by maintaining the number of optimal solutions and increasing the number of suboptimal solutions generated. Also, providing more information to the model improves the results, both by adding attributes to the objects, and by increasing the number of objects. The reranking system based on Object bounding box area gets the best results, reinforcing the idea that the questions focus on objects clearly represented in the image. |
|---|