Large language models surpass human experts in predicting neuroscience results
[Data availability] Human participant data, and intermediate data generated via simulations and analyses, are publicly available via GitHub at https://github.com/braingpt-lovelab/BrainBench. Model weights and training data are available at https://huggingface.co/BrainGPT. Model training data are sou...
| Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | article |
| Status: | Published version |
| Publication Date: | 2025 |
| Country: | España |
| Institution: | Consejo Superior de Investigaciones Científicas (CSIC) |
| Repository: | DIGITAL.CSIC. Repositorio Institucional del CSIC |
| OAI Identifier: | oai:digital.csic.es:10261/395833 |
| Online Access: | http://hdl.handle.net/10261/395833 https://api.elsevier.com/content/abstract/scopus_id/85203189734 |
| Access Level: | Open access |
| Keyword: | Neuroscience Scientific community http://metadata.un.org/sdg/3 Ensure healthy lives and promote well-being for all at all ages |
| id |
ES_e3576e2efe321ae0a575201ae750a596 |
|---|---|
| oai_identifier_str |
oai:digital.csic.es:10261/395833 |
| network_acronym_str |
ES |
| network_name_str |
España |
| repository_id_str |
|
| dc.title.none.fl_str_mv |
Large language models surpass human experts in predicting neuroscience results |
| title |
Large language models surpass human experts in predicting neuroscience results |
| spellingShingle |
Large language models surpass human experts in predicting neuroscience results Luo, Xiaoliang Neuroscience Scientific community http://metadata.un.org/sdg/3 Ensure healthy lives and promote well-being for all at all ages |
| title_short |
Large language models surpass human experts in predicting neuroscience results |
| title_full |
Large language models surpass human experts in predicting neuroscience results |
| title_fullStr |
Large language models surpass human experts in predicting neuroscience results |
| title_full_unstemmed |
Large language models surpass human experts in predicting neuroscience results |
| title_sort |
Large language models surpass human experts in predicting neuroscience results |
| dc.creator.none.fl_str_mv |
Luo, Xiaoliang Rechardt, Akilles Sun, Guangzhi Nejad, Kevin K Yáñez, Felipe Yilmaz, Bati Lee, Kangjoo Cohen, Alexandra O Borghesani, Valentina Pashkov, Anton Marinazzo, Daniele Nicholas, Jonathan Salatiello, Alessandro Sucholutsky, Ilia Minervini, Pasquale Razavi, Sepehr Rocca, Roberta Yusifov, Elkhan Okalova, Tereza Gu, Nianlong Ferianc, Martin Khona, Mikail Patil, Kaustubh R Lee, Pui-Shee Mata, Rui Myers, Nicholas E Bizley, Jennifer K Musslick, Sebastian Bilgin, Isil Poyraz Niso, Guiomar Ales, Justin M Gaebler, Michael Ratan Murty, N Apurva Loued-Khenissi, Leyla Behler, Anna Hall, Chloe M Dafflon, Jessica Bao, Sherry Dongqi Love, Bradley C |
| author |
Luo, Xiaoliang |
| author_facet |
Luo, Xiaoliang Rechardt, Akilles Sun, Guangzhi Nejad, Kevin K Yáñez, Felipe Yilmaz, Bati Lee, Kangjoo Cohen, Alexandra O Borghesani, Valentina Pashkov, Anton Marinazzo, Daniele Nicholas, Jonathan Salatiello, Alessandro Sucholutsky, Ilia Minervini, Pasquale Razavi, Sepehr Rocca, Roberta Yusifov, Elkhan Okalova, Tereza Gu, Nianlong Ferianc, Martin Khona, Mikail Patil, Kaustubh R Lee, Pui-Shee Mata, Rui Myers, Nicholas E Bizley, Jennifer K Musslick, Sebastian Bilgin, Isil Poyraz Niso, Guiomar Ales, Justin M Gaebler, Michael Ratan Murty, N Apurva Loued-Khenissi, Leyla Behler, Anna Hall, Chloe M Dafflon, Jessica Bao, Sherry Dongqi Love, Bradley C |
| author_role |
author |
| author2 |
Rechardt, Akilles Sun, Guangzhi Nejad, Kevin K Yáñez, Felipe Yilmaz, Bati Lee, Kangjoo Cohen, Alexandra O Borghesani, Valentina Pashkov, Anton Marinazzo, Daniele Nicholas, Jonathan Salatiello, Alessandro Sucholutsky, Ilia Minervini, Pasquale Razavi, Sepehr Rocca, Roberta Yusifov, Elkhan Okalova, Tereza Gu, Nianlong Ferianc, Martin Khona, Mikail Patil, Kaustubh R Lee, Pui-Shee Mata, Rui Myers, Nicholas E Bizley, Jennifer K Musslick, Sebastian Bilgin, Isil Poyraz Niso, Guiomar Ales, Justin M Gaebler, Michael Ratan Murty, N Apurva Loued-Khenissi, Leyla Behler, Anna Hall, Chloe M Dafflon, Jessica Bao, Sherry Dongqi Love, Bradley C |
| author2_role |
author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author author |
| dc.contributor.none.fl_str_mv |
Economic and Social Research Council (UK) Microsoft 0000-0002-5297-2114 0009-0009-9991-4577 0000-0002-5886-056X 0009-0006-4441-5517 0000-0002-9598-0616 0000-0002-8824-0345 0000-0002-7760-8079 0000-0002-3116-3529 #NODATA# #NODATA# 0000-0002-9803-0122 #NODATA# 0000-0003-2311-8645 0000-0003-4121-7479 0000-0002-8442-602X 0000-0003-4651-8852 0000-0001-9017-8088 0000-0002-7227-839X 0009-0009-1688-6655 0000-0002-8474-0836 0000-0002-4031-6398 #NODATA# 0000-0002-0289-5480 0000-0002-8016-9468 0000-0002-1679-906X 0000-0001-5599-3044 0000-0001-6605-2362 0000-0002-8896-639X #NODATA# 0000-0001-5872-8924 0000-0002-2089-1563 0000-0002-4442-5778 0000-0003-2191-797X 0000-0003-3241-6492 0000-0001-7129-7006 #NODATA# 0000-0003-2540-0927 0000-0003-4804-3379 0000-0002-7883-7076 Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72] |
| dc.subject.none.fl_str_mv |
Neuroscience Scientific community http://metadata.un.org/sdg/3 Ensure healthy lives and promote well-being for all at all ages |
| topic |
Neuroscience Scientific community http://metadata.un.org/sdg/3 Ensure healthy lives and promote well-being for all at all ages |
| description |
[Data availability] Human participant data, and intermediate data generated via simulations and analyses, are publicly available via GitHub at https://github.com/braingpt-lovelab/BrainBench. Model weights and training data are available at https://huggingface.co/BrainGPT. Model training data are sourced from PubMed and PubMed Central Open Access Subset (PMC OAS) using the Entrez Programming Utilities (E-utilities) API and the pubget Python package, respectively. |
| publishDate |
2025 |
| dc.date.none.fl_str_mv |
2025 2025 2025 |
| dc.type.none.fl_str_mv |
info:eu-repo/semantics/article http://purl.org/coar/resource_type/c_6501 Publisher's version info:eu-repo/semantics/publishedVersion |
| format |
article |
| status_str |
publishedVersion |
| dc.identifier.none.fl_str_mv |
http://hdl.handle.net/10261/395833 https://api.elsevier.com/content/abstract/scopus_id/85203189734 |
| url |
http://hdl.handle.net/10261/395833 https://api.elsevier.com/content/abstract/scopus_id/85203189734 |
| dc.language.none.fl_str_mv |
Inglés |
| language_invalid_str_mv |
Inglés |
| dc.relation.none.fl_str_mv |
The underlying dataset has been published as supplementary material of the article in the publisher platform at DOI https://doi.org/10.1038/s41562-024-02046-9 https://doi.org/10.1038/s41562-024-02046-9 Sí |
| dc.rights.none.fl_str_mv |
info:eu-repo/semantics/openAccess |
| eu_rights_str_mv |
openAccess |
| dc.format.none.fl_str_mv |
application/pdf |
| dc.publisher.none.fl_str_mv |
Nature Publishing Group |
| publisher.none.fl_str_mv |
Nature Publishing Group |
| dc.source.none.fl_str_mv |
reponame:DIGITAL.CSIC. Repositorio Institucional del CSIC instname:Consejo Superior de Investigaciones Científicas (CSIC) |
| instname_str |
Consejo Superior de Investigaciones Científicas (CSIC) |
| reponame_str |
DIGITAL.CSIC. Repositorio Institucional del CSIC |
| collection |
DIGITAL.CSIC. Repositorio Institucional del CSIC |
| repository.name.fl_str_mv |
|
| repository.mail.fl_str_mv |
|
| _version_ |
1869422493059514368 |
| spelling |
Large language models surpass human experts in predicting neuroscience resultsLuo, XiaoliangRechardt, AkillesSun, GuangzhiNejad, Kevin KYáñez, FelipeYilmaz, BatiLee, KangjooCohen, Alexandra OBorghesani, ValentinaPashkov, AntonMarinazzo, DanieleNicholas, JonathanSalatiello, AlessandroSucholutsky, IliaMinervini, PasqualeRazavi, SepehrRocca, RobertaYusifov, ElkhanOkalova, TerezaGu, NianlongFerianc, MartinKhona, MikailPatil, Kaustubh RLee, Pui-SheeMata, RuiMyers, Nicholas EBizley, Jennifer KMusslick, SebastianBilgin, Isil PoyrazNiso, GuiomarAles, Justin MGaebler, MichaelRatan Murty, N ApurvaLoued-Khenissi, LeylaBehler, AnnaHall, Chloe MDafflon, JessicaBao, Sherry DongqiLove, Bradley CNeuroscienceScientific communityhttp://metadata.un.org/sdg/3Ensure healthy lives and promote well-being for all at all ages[Data availability] Human participant data, and intermediate data generated via simulations and analyses, are publicly available via GitHub at https://github.com/braingpt-lovelab/BrainBench. Model weights and training data are available at https://huggingface.co/BrainGPT. Model training data are sourced from PubMed and PubMed Central Open Access Subset (PMC OAS) using the Entrez Programming Utilities (E-utilities) API and the pubget Python package, respectively.[Code availability] All computer code associated with this work including model training, evaluation, data processing and analyses are publicly available via GitHub at https://github.com/braingpt-lovelab/BrainBench.Scientific discoveries often hinge on synthesizing decades of research, a task that potentially outstrips human information processing capacities. Large language models (LLMs) offer a solution. LLMs trained on the vast scientific literature could potentially integrate noisy yet interrelated findings to forecast novel results better than human experts. Here, to evaluate this possibility, we created BrainBench, a forward-looking benchmark for predicting neuroscience results. We find that LLMs surpass experts in predicting experimental outcomes. BrainGPT, an LLM we tuned on the neuroscience literature, performed better yet. Like human experts, when LLMs indicated high confidence in their predictions, their responses were more likely to be correct, which presages a future where LLMs assist humans in making discoveries. Our approach is not neuroscience specific and is transferable to other knowledge-intensive endeavours.This work was supported the ESRC (ES/W007347/1), Microsoft (Accelerate Foundation Models Research Program) and a Royal Society Wolfson Fellowship (18302) to B.C.L. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript. We thank M. Garvert, P. R. Raamana, T. Hare, Y. Kessler, O. Robinson and D.R. for their assistance. We thank the 202 participants of the online study, including: G. Molinaro, J. Zhu, M. Abdallah, Y. G. Pavlov, J. Lee, A. Harris, Z. Li, R. Kessler, L. Zhang, M. Szul, P. Gupta, S. Bhattacharya, J. Prinsen, C. Gallagher, M. Anes, M. Laroy, T. Ackels, C. Forster, P. Gonçalves, T. Mcconnell, D. Whitmer, D. Kundu, B. Pasquereau, J. Manning, M. Szul, A. Hussain, N. Clairis, I., Vega-Vásquez, K. Chen, J. Hogeveen, S. Salehi, S. Duraivel, E. Guevara, Z. Zhang, T. J. Younts, M. Muszyński, L. Dalla Porta, T. Gureckis, P. Rafei, F.-C. Chou, K. Temple, A. Altunkaya, A. Tan, J. H. Yun, A. Marin-Llobet, B. Lord, D. Lindh, S. Besson-Girard, E. Irmak, E. Çelik, A. Maharjan and I. S. Plank.Peer reviewedNature Publishing GroupEconomic and Social Research Council (UK)Microsoft0000-0002-5297-21140009-0009-9991-45770000-0002-5886-056X0009-0006-4441-55170000-0002-9598-06160000-0002-8824-03450000-0002-7760-80790000-0002-3116-3529#NODATA##NODATA#0000-0002-9803-0122#NODATA#0000-0003-2311-86450000-0003-4121-74790000-0002-8442-602X0000-0003-4651-88520000-0001-9017-80880000-0002-7227-839X0009-0009-1688-66550000-0002-8474-08360000-0002-4031-6398#NODATA#0000-0002-0289-54800000-0002-8016-94680000-0002-1679-906X0000-0001-5599-30440000-0001-6605-23620000-0002-8896-639X#NODATA#0000-0001-5872-89240000-0002-2089-15630000-0002-4442-57780000-0003-2191-797X0000-0003-3241-64920000-0001-7129-7006#NODATA#0000-0003-2540-09270000-0003-4804-33790000-0002-7883-7076Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72]202520252025info:eu-repo/semantics/articlehttp://purl.org/coar/resource_type/c_6501Publisher's versioninfo:eu-repo/semantics/publishedVersionapplication/pdfhttp://hdl.handle.net/10261/395833https://api.elsevier.com/content/abstract/scopus_id/85203189734reponame:DIGITAL.CSIC. Repositorio Institucional del CSICinstname:Consejo Superior de Investigaciones Científicas (CSIC)InglésThe underlying dataset has been published as supplementary material of the article in the publisher platform at DOI https://doi.org/10.1038/s41562-024-02046-9https://doi.org/10.1038/s41562-024-02046-9Síinfo:eu-repo/semantics/openAccessoai:digital.csic.es:10261/3958332026-05-22T06:33:51Z |
| score |
15.811543 |