General scales unlock AI evaluation with explanatory and predictive power
[EN] Ensuring safe and effective use of artificial intelligence (AI) requires understanding and anticipating its performance on new tasks, from advanced scientific challenges to transformed workplace activities1, 2-3. So far, benchmarking has guided progress in AI but has offered limited explanatory...
| Autores: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2026 |
| País: | España |
| Institución: | Universitat Politècnica de València (UPV) |
| Repositorio: | RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia |
| Idioma: | inglés |
| OAI Identifier: | oai:dnet:riunet______::40f6186df3ba6a32dfa9512302a4ae2b |
| Acceso en línea: | https://riunet.upv.es/handle/10251/235051 |
| Access Level: | acceso abierto |
| Palabra clave: | Benchmarking methods Artificial intelligence evaluation Large language models Performance prediction Ability profiling Out-of-distribution generalization |
| Sumario: | [EN] Ensuring safe and effective use of artificial intelligence (AI) requires understanding and anticipating its performance on new tasks, from advanced scientific challenges to transformed workplace activities1, 2-3. So far, benchmarking has guided progress in AI but has offered limited explanatory and predictive power for general-purpose AI systems4, 5, 6, 7-8, attributed to limited transferability across specific tasks9, 10-11. Here we introduce general scales for AI evaluation that elicit demand profiles explaining what capabilities common AI benchmarks truly measure, extract ability profiles quantifying the general strengths and limits of AI systems and robustly predict AI performance for new task instances. Our fully automated methodology builds on 18 rubrics, capturing a broad range of cognitive and intellectual demands, which place different task instances on the same general scales, illustrated on 15 large language models (LLMs) and 63 tasks. Both the demand and the ability profiles on these scales bring new insights such as construct validity through benchmark sensitivity and specificity and explain conflicting claims about whether AI has reasoning capabilities. Ultimately, high predictive power at the instance level becomes possible using the general scales, providing superior estimates over strong black-box baseline predictors, especially in out-of-distribution settings (new tasks and benchmarks). The scales, rubrics, battery, techniques and results presented here constitute a solid foundation for a science of AI evaluation, underpinning the reliable deployment of AI in the years ahead. |
|---|