MultiSynDS: large scale multilingual synthetic discharge summary generation by means of generative LLMs and clinical case report transformation
Access to real clinical records is often restricted because of privacy and legal constraints, particularly for languages other than English. This limitation reduces the availability of structured data needed to develop and evaluate multilingual clinical NLP systems. This thesis introduces MultiSynDS...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2025 |
| País: | España |
| Institución: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/446598 |
| Acceso en línea: | https://hdl.handle.net/2117/446598 |
| Access Level: | acceso abierto |
| Palabra clave: | Medical records Natural language processing (Computer science) Datos sintéticos PLN clínico PLN multilingüe Informes de alta Modelos de lenguaje a gran escala Evaluación de modelos Generación de lenguaje natural LLM como juez Synthetic data Clinical NLP Multilingual NLP Discharge summaries Large language models Model evaluation Natural Language Generation LLM as a Judge Històries clíniques Tractament del llenguatge natural (Informàtica) Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural |
| Sumario: | Access to real clinical records is often restricted because of privacy and legal constraints, particularly for languages other than English. This limitation reduces the availability of structured data needed to develop and evaluate multilingual clinical NLP systems. This thesis introduces MultiSynDS, a pipeline that generates synthetic, multilingual discharge summaries from publicly available clinical case reports. Leveraging generative large language models, the approach transforms free-text narratives into structured summaries and produces translations in multiple languages. The investigation is organized around five research questions: first, whether a generative model can produce clinically acceptable discharge summaries; second, which factors complicate human evaluation of these summaries; third, how well concept-level automatic metrics align with expert judgments; fourth, the utility of the LLM-as-a-judge method and the influence of model size and domain adaptation; and fifth, whether models adapted to the medical domain improve agreement with expert evaluations compared to general-purpose models. The method was applied to 1,000 cardiology case reports from the AI4HF project. Synthetic summaries were assessed via clinical named-entity recognition, SNOMED CT concept linking, expert reviews on a 1¿5 scale, and rubric-based scoring by LLMs. For 30 selected summary pairs, human annotators assigned mean ratings of 4.09 for medical entities completeness, 4.49 for structure-header, 4.02 for structure-content, 4.21 for content accuracy, 4.25 for made-up content detection, and an overall quality score of 3.63. The findings suggest that concept-based automatic metrics correlate well with expert evaluations on core content dimensions, whereas both experts and LLMs exhibit lower agreement on more subjective aspects such as hallucinations and overall score. Although this work focuses on English, the pipeline is inherently multilingual and can be extended to other languages included in the DataTools4Heart project-namely Spanish, Dutch, Italian, Romanian, Czech, and Swedish. The full MultiSynDS dataset, comprising original clinical cases and generated discharge summaries in English, Spanish, and Dutch, is publicly available at https://zenodo.org/records/15664866. The codebase is also shared with the open-source community on GitHub at https://github.com/nlp4bia-bsc/MultiSynDS. |
|---|