Evaluating the Performance of DeepSeek 3, Claude Sonnet 4, and Gemini 2.5 in the Chilean Medical Licensing Examination: Observational Study.
Introduction: Artificial intelligences and their continuous improvement have revolutionized medical education, but their performance in specific evaluative contexts still requires further exploration. Methods: This study qualitatively evaluated and compared the performance of three state-of-the-art...
| Autores: | , , , , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2025 |
| País: | España |
| Institución: | Universidad de Murcia |
| Repositorio: | DIGITUM. Depósito Digital Institucional de la Universidad de Murcia |
| OAI Identifier: | oai:digitum.um.es:10201/201202 |
| Acceso en línea: | https://doi.org/10.6018/edumed.679731 http://hdl.handle.net/10201/201202 |
| Access Level: | acceso abierto |
| Palabra clave: | Medical education EUNACOM Clinical reasoning Language models Medical assessment Inteligencia artificial Educación médica Razonamiento clínico Modelos de lenguaje Evaluación médica Artificial intelligence No relacionado con ningún objetivo de desarrollo sostenible |
| Sumario: | Introduction: Artificial intelligences and their continuous improvement have revolutionized medical education, but their performance in specific evaluative contexts still requires further exploration. Methods: This study qualitatively evaluated and compared the performance of three state-of-the-art language models — Claude Sonnet 4, Gemini 2.5, and DeepSeek 3 — in simulations of the National Medical Knowledge Examination (EUNACOM) in Chile. Three mock exams with 180 questions each were used, covering various medical areas and question types, including those based on clinical cases. Results: The results show that all AI models consistently passed the exams, with Claude Sonnet 4 achieving the highest overall performance (89% accuracy) and the greatest consistency across attempts. Clinical case-based questions were answered more accurately than theoretical knowledge questions, highlighting the models' strength in contextual clinical reasoning. Claude excelled in Internal Medicine and Psychiatry, DeepSeek in Surgery, and Gemini demonstrated balanced performance. However, specific gaps were identified in areas such as Public Health and clinical follow-up, suggesting the need for model-specific adjustments. Conclusion: The findings support the educational potential of these tools but also emphasize the importance of their ethical, supervised, and complementary use alongside traditional medical training. This study contributes to understanding the emerging role of artificial intelligence in professional assessments, as well as its limitations and opportunities within the Chilean medical context. |
|---|