Evaluating LLMs’ abilities to create charts, a systematic approach

The use of generative models, especially those based on pretrained transformers, has become a common practice in code development. Tools such as GitHub Copilot, Cursor, and the direct use of conversational chatbots have proven useful to accelerate the development of applications. Unfortunately, gene...

Descripción completa

Detalles Bibliográficos
Autores: Ribalta Albado, Maria, Vázquez Alcocer, Pere Pau|||0000-0003-4638-4065
Tipo de recurso: artículo
Fecha de publicación:2026
País:España
Institución:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/459319
Acceso en línea:https://hdl.handle.net/2117/459319
https://dx.doi.org/10.1016/j.cag.2026.104544
Access Level:acceso abierto
Palabra clave:Visualization
Large language models
Evaluation
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
Àrees temàtiques de la UPC::Informàtica::Infografia
id ES_ed01c2e3b0835558d38d6aa30e6aedfd
oai_identifier_str oai:upcommons.upc.edu:2117/459319
network_acronym_str ES
network_name_str España
repository_id_str
spelling Evaluating LLMs’ abilities to create charts, a systematic approachRibalta Albado, MariaVázquez Alcocer, Pere Pau|||0000-0003-4638-4065VisualizationLarge language modelsEvaluationÀrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge naturalÀrees temàtiques de la UPC::Informàtica::InfografiaThe use of generative models, especially those based on pretrained transformers, has become a common practice in code development. Tools such as GitHub Copilot, Cursor, and the direct use of conversational chatbots have proven useful to accelerate the development of applications. Unfortunately, generative models are unable to determine what is correct or wrong, and their outputs may contain errors. Their stochastic nature does not guarantee a single solution for the same problem, either. Furthermore, the output depends largely on the prompt issued by the user. To assess the capabilities of LLMs, some benchmarks have been proposed. Unfortunately, they often rely on ground truth data that may not be available. As a result, the extent to which modern LLMs can create charts needs further investigation. This work contributes to the understanding of the generative models’ ability to create charts in three ways: (a) Creating a dataset of prompts, data sources, and chart types to analyze, (b) Designing a set of systematic experiments that cover a wide range of commonly used charts, and variations of the visual variables, and (c) by empirically analyzing the performance of a large set of LLMs of different sizes, including Claude, CodeLlama, Gemini, Gemma, GPT4o, Llama 3.1, and Mixtral. Our results indicate that even the most advanced LLMs have room for improvement.This project has been supported by PID2021-122136OB-C21 from the Ministerio de Ciencia e Innovación, Spain, by 839 FEDER (EU) funds, and 2021 SGR 01035 by Generalitat de Catalunya, Spain.Peer ReviewedElsevier20262026-04-0120262026-03-24journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/459319https://dx.doi.org/10.1016/j.cag.2026.104544reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)InglésengAgencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-122136OB-C21 ENTORNOS 3D DE ALTA FIDELIDAD PARA REALIDAD VIRTUAL Y COMPUTACION VISUAL: GEOMETRIA, MOVIMIENTO, INTERACCION Y VISUALIZACION PARA SALUD, ARQUITECTURA Y CIUDADESopen accesshttp://purl.org/coar/access_right/c_abf2Attribution-NonCommercial 4.0 Internationalhttp://creativecommons.org/licenses/by-nc/4.0/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4593192026-05-27T15:37:01Z
dc.title.none.fl_str_mv Evaluating LLMs’ abilities to create charts, a systematic approach
title Evaluating LLMs’ abilities to create charts, a systematic approach
spellingShingle Evaluating LLMs’ abilities to create charts, a systematic approach
Ribalta Albado, Maria
Visualization
Large language models
Evaluation
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
Àrees temàtiques de la UPC::Informàtica::Infografia
title_short Evaluating LLMs’ abilities to create charts, a systematic approach
title_full Evaluating LLMs’ abilities to create charts, a systematic approach
title_fullStr Evaluating LLMs’ abilities to create charts, a systematic approach
title_full_unstemmed Evaluating LLMs’ abilities to create charts, a systematic approach
title_sort Evaluating LLMs’ abilities to create charts, a systematic approach
dc.creator.none.fl_str_mv Ribalta Albado, Maria
Vázquez Alcocer, Pere Pau|||0000-0003-4638-4065
author Ribalta Albado, Maria
author_facet Ribalta Albado, Maria
Vázquez Alcocer, Pere Pau|||0000-0003-4638-4065
author_role author
author2 Vázquez Alcocer, Pere Pau|||0000-0003-4638-4065
author2_role author
dc.subject.none.fl_str_mv Visualization
Large language models
Evaluation
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
Àrees temàtiques de la UPC::Informàtica::Infografia
topic Visualization
Large language models
Evaluation
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
Àrees temàtiques de la UPC::Informàtica::Infografia
description The use of generative models, especially those based on pretrained transformers, has become a common practice in code development. Tools such as GitHub Copilot, Cursor, and the direct use of conversational chatbots have proven useful to accelerate the development of applications. Unfortunately, generative models are unable to determine what is correct or wrong, and their outputs may contain errors. Their stochastic nature does not guarantee a single solution for the same problem, either. Furthermore, the output depends largely on the prompt issued by the user. To assess the capabilities of LLMs, some benchmarks have been proposed. Unfortunately, they often rely on ground truth data that may not be available. As a result, the extent to which modern LLMs can create charts needs further investigation. This work contributes to the understanding of the generative models’ ability to create charts in three ways: (a) Creating a dataset of prompts, data sources, and chart types to analyze, (b) Designing a set of systematic experiments that cover a wide range of commonly used charts, and variations of the visual variables, and (c) by empirically analyzing the performance of a large set of LLMs of different sizes, including Claude, CodeLlama, Gemini, Gemma, GPT4o, Llama 3.1, and Mixtral. Our results indicate that even the most advanced LLMs have room for improvement.
publishDate 2026
dc.date.none.fl_str_mv 2026
2026-04-01
2026
2026-03-24
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/459319
https://dx.doi.org/10.1016/j.cag.2026.104544
url https://hdl.handle.net/2117/459319
https://dx.doi.org/10.1016/j.cag.2026.104544
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.relation.none.fl_str_mv Agencia Estatal de Investigación http://doi.org/10.13039/501100011033 Plan Estatal de Investigación Científica y Técnica y de Innovación 2021-2023 PID2021-122136OB-C21 ENTORNOS 3D DE ALTA FIDELIDAD PARA REALIDAD VIRTUAL Y COMPUTACION VISUAL: GEOMETRIA, MOVIMIENTO, INTERACCION Y VISUALIZACION PARA SALUD, ARQUITECTURA Y CIUDADES
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial 4.0 International
http://creativecommons.org/licenses/by-nc/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution-NonCommercial 4.0 International
http://creativecommons.org/licenses/by-nc/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier
publisher.none.fl_str_mv Elsevier
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869423390184439808
score 15,812429

Similares en LA Referencia