Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study.
Introduction: Artificial intelligences and their continuous improvement have revolutionized medical education, but their performance in specific evaluative contexts still requires further exploration. Methods: This study qualitatively evaluated and compared the performance of three state-of-the-art...
| Autores: | , , , , , |
|---|---|
| Tipo de documento: | artigo |
| Data de publicação: | 2025 |
| País: | España |
| Recursos: | Universidad de Murcia |
| Repositório: | DIGITUM. Depósito Digital Institucional de la Universidad de Murcia |
| OAI Identifier: | oai:digitum.um.es:10201/201202 |
| Acesso em linha: | https://doi.org/10.6018/edumed.679731 http://hdl.handle.net/10201/201202 |
| Access Level: | Acceso aberto |
| Palavra-chave: | Medical education EUNACOM Clinical reasoning Language models Medical assessment Inteligencia artificial Educación médica Razonamiento clínico Modelos de lenguaje Evaluación médica Artificial intelligence No relacionado con ningún objetivo de desarrollo sostenible |
| id |
ES_2506e9fb55fd0fdbf2ecb5abd93eb411 |
|---|---|
| oai_identifier_str |
oai:digitum.um.es:10201/201202 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| dc.title.none.fl_str_mv |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. Evaluación del desempeño de DeepSeek 3, Claude Sonnet 4 y Gemini 2.5 en el examen de licencia médica chileno: estudio observacional. |
| title |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. |
| spellingShingle |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. Jerez Yañez, Oscar Medical education EUNACOM Clinical reasoning Language models Medical assessment Inteligencia artificial Educación médica Razonamiento clínico Modelos de lenguaje Evaluación médica Artificial intelligence No relacionado con ningún objetivo de desarrollo sostenible |
| title_short |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. |
| title_full |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. |
| title_fullStr |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. |
| title_full_unstemmed |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. |
| title_sort |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study. |
| dc.creator.none.fl_str_mv |
Jerez Yañez, Oscar Edgardo, Vicente Alberto Silva Arroyo, Jesús Vera Cartes, Marcos Jeremías Giovanny Herrera Alcaíno, Alvaro Andrés Lancellotti Guajardo, Anaís Aracelly |
| author |
Jerez Yañez, Oscar |
| author_facet |
Jerez Yañez, Oscar Edgardo, Vicente Alberto Silva Arroyo, Jesús Vera Cartes, Marcos Jeremías Giovanny Herrera Alcaíno, Alvaro Andrés Lancellotti Guajardo, Anaís Aracelly |
| author_role |
author |
| author2 |
Edgardo, Vicente Alberto Silva Arroyo, Jesús Vera Cartes, Marcos Jeremías Giovanny Herrera Alcaíno, Alvaro Andrés Lancellotti Guajardo, Anaís Aracelly |
| author2_role |
author author author author author |
| dc.contributor.none.fl_str_mv |
Sin departamento asociado |
| dc.subject.none.fl_str_mv |
Medical education EUNACOM Clinical reasoning Language models Medical assessment Inteligencia artificial Educación médica Razonamiento clínico Modelos de lenguaje Evaluación médica Artificial intelligence No relacionado con ningún objetivo de desarrollo sostenible |
| topic |
Medical education EUNACOM Clinical reasoning Language models Medical assessment Inteligencia artificial Educación médica Razonamiento clínico Modelos de lenguaje Evaluación médica Artificial intelligence No relacionado con ningún objetivo de desarrollo sostenible |
| description |
Introduction: Artificial intelligences and their continuous improvement have revolutionized medical education, but their performance in specific evaluative contexts still requires further exploration. Methods: This study qualitatively evaluated and compared the performance of three state-of-the-art language models — Claude Sonnet 4, Gemini 2.5, and DeepSeek 3 — in simulations of the National Medical Knowledge Examination (EUNACOM) in Chile. Three mock exams with 180 questions each were used, covering various medical areas and question types, including those based on clinical cases. Results: The results show that all AI models consistently passed the exams, with Claude Sonnet 4 achieving the highest overall performance (89% accuracy) and the greatest consistency across attempts. Clinical case-based questions were answered more accurately than theoretical knowledge questions, highlighting the models' strength in contextual clinical reasoning. Claude excelled in Internal Medicine and Psychiatry, DeepSeek in Surgery, and Gemini demonstrated balanced performance. However, specific gaps were identified in areas such as Public Health and clinical follow-up, suggesting the need for model-specific adjustments. Conclusion: The findings support the educational potential of these tools but also emphasize the importance of their ethical, supervised, and complementary use alongside traditional medical training. This study contributes to understanding the emerging role of artificial intelligence in professional assessments, as well as its limitations and opportunities within the Chilean medical context. |
| publishDate |
2025 |
| dc.date.none.fl_str_mv |
2025 2026 2026 |
| dc.type.none.fl_str_mv |
info:eu-repo/semantics/article |
| format |
article |
| dc.identifier.none.fl_str_mv |
https://doi.org/10.6018/edumed.679731 http://hdl.handle.net/10201/201202 |
| url |
https://doi.org/10.6018/edumed.679731 http://hdl.handle.net/10201/201202 |
| dc.language.none.fl_str_mv |
Inglés |
| language_invalid_str_mv |
Inglés |
| dc.relation.none.fl_str_mv |
Sin financiación externa a la Universidad |
| dc.rights.none.fl_str_mv |
Attribution-NonCommercial-NoDerivatives 4.0 International info:eu-repo/semantics/openAccess http://creativecommons.org/licenses/by-nc-nd/4.0/ |
| rights_invalid_str_mv |
Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf 9 application/pdf |
| dc.publisher.none.fl_str_mv |
Servicio de Publicaciones. Universidad de Murcia |
| publisher.none.fl_str_mv |
Servicio de Publicaciones. Universidad de Murcia |
| dc.source.none.fl_str_mv |
reponame:DIGITUM. Depósito Digital Institucional de la Universidad de Murcia instname:Universidad de Murcia |
| instname_str |
Universidad de Murcia |
| reponame_str |
DIGITUM. Depósito Digital Institucional de la Universidad de Murcia |
| collection |
DIGITUM. Depósito Digital Institucional de la Universidad de Murcia |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869404733545906176 |
| spelling |
Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study.Evaluación del desempeño de DeepSeek 3, Claude Sonnet 4 y Gemini 2.5 en el examen de licencia médica chileno: estudio observacional.Jerez Yañez, OscarEdgardo, Vicente AlbertoSilva Arroyo, JesúsVera Cartes, Marcos Jeremías GiovannyHerrera Alcaíno, Alvaro AndrésLancellotti Guajardo, Anaís AracellyMedical educationEUNACOMClinical reasoningLanguage modelsMedical assessmentInteligencia artificialEducación médicaRazonamiento clínicoModelos de lenguajeEvaluación médicaArtificial intelligenceNo relacionado con ningún objetivo de desarrollo sostenibleIntroduction: Artificial intelligences and their continuous improvement have revolutionized medical education, but their performance in specific evaluative contexts still requires further exploration. Methods: This study qualitatively evaluated and compared the performance of three state-of-the-art language models — Claude Sonnet 4, Gemini 2.5, and DeepSeek 3 — in simulations of the National Medical Knowledge Examination (EUNACOM) in Chile. Three mock exams with 180 questions each were used, covering various medical areas and question types, including those based on clinical cases. Results: The results show that all AI models consistently passed the exams, with Claude Sonnet 4 achieving the highest overall performance (89% accuracy) and the greatest consistency across attempts. Clinical case-based questions were answered more accurately than theoretical knowledge questions, highlighting the models' strength in contextual clinical reasoning. Claude excelled in Internal Medicine and Psychiatry, DeepSeek in Surgery, and Gemini demonstrated balanced performance. However, specific gaps were identified in areas such as Public Health and clinical follow-up, suggesting the need for model-specific adjustments. Conclusion: The findings support the educational potential of these tools but also emphasize the importance of their ethical, supervised, and complementary use alongside traditional medical training. This study contributes to understanding the emerging role of artificial intelligence in professional assessments, as well as its limitations and opportunities within the Chilean medical context.La inteligencias artificial y su mejora continua han revolucionado la educación médica, pero su desempeño en contextos evaluativos específicos aún requiere mayor exploración. Métodos: Este estudio evaluó y comparó cualitativamente el desempeño de tres modelos de lenguaje de última generación —Claude Sonnet 4, Gemini 2.5 y DeepSeek 3— en simulaciones del Examen Nacional de Conocimientos Médicos (EUNACOM) en Chile. Se utilizaron tres exámenes simulados con 180 preguntas cada uno, que abarcaban diversas áreas médicas y tipos de preguntas, incluidas las basadas en casos clínicos. Resultados: Los resultados muestran que todos los modelos de IA aprobaron los exámenes de forma consistente, y Claude Sonnet 4 logró el mayor desempeño general (89% de precisión) y la mayor consistencia en todos los intentos. Las preguntas basadas en casos clínicos se respondieron con mayor precisión que las preguntas de conocimiento teórico, lo que destaca la fortaleza de los modelos en el razonamiento clínico contextual. Claude sobresalió en Medicina Interna y Psiquiatría, DeepSeek en Cirugía y Gemini demostró un desempeño equilibrado. Sin embargo, se identificaron deficiencias específicas en áreas como la salud pública y el seguimiento clínico, lo que sugiere la necesidad de realizar ajustes específicos a cada modelo. Conclusión: Los hallazgos respaldan el potencial educativo de estas herramientas, pero también enfatizan la importancia de su uso ético, supervisado y complementario a la formación médica tradicional. Este estudio contribuye a comprender el papel emergente de la inteligencia artificial en las evaluaciones profesionales, así como sus limitaciones y oportunidades en el contexto médico chileno.Servicio de Publicaciones. Universidad de MurciaSin departamento asociado202620262025info:eu-repo/semantics/articleapplication/pdf9application/pdfhttps://doi.org/10.6018/edumed.679731http://hdl.handle.net/10201/201202reponame:DIGITUM. Depósito Digital Institucional de la Universidad de Murciainstname:Universidad de MurciaInglésSin financiación externa a la UniversidadAttribution-NonCommercial-NoDerivatives 4.0 Internationalinfo:eu-repo/semantics/openAccesshttp://creativecommons.org/licenses/by-nc-nd/4.0/oai:digitum.um.es:10201/2012022026-05-27T12:40:41Z |
| score |
15,811543 |