Improving reliability estimation in cognitive diagnosis modeling

Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability e...

Descripción completa

Detalles Bibliográficos
Autores: Schames Kreitchmann, Rodrigo, de la Torre, Jimmy, Sorrel Luján, Miguel Ángel, Najera Álvarez, Pablo, Abad García, Francisco José
Tipo de recurso: artículo
Fecha de publicación:2022
País:España
Institución:Universidad Autónoma de Madrid
Repositorio:Biblos-e Archivo. Repositorio Institucional de la UAM
Idioma:inglés
OAI Identifier:oai:repositorio.uam.es:10486/704451
Acceso en línea:http://hdl.handle.net/10486/704451
https://dx.doi.org/10.3758/s13428-022-01967-5
Access Level:acceso abierto
Palabra clave:Classification accuracy
Cognitive diagnosis
Diagnostic classification
Multiple imputation
Reliability
Psicología
id ES_3ff0dcb8097726c33c2d1638effd6cc3
oai_identifier_str oai:repositorio.uam.es:10486/704451
network_acronym_str ES
network_name_str España
repository_id_str
spelling Improving reliability estimation in cognitive diagnosis modelingSchames Kreitchmann, Rodrigode la Torre, JimmySorrel Luján, Miguel ÁngelNajera Álvarez, PabloAbad García, Francisco JoséClassification accuracyCognitive diagnosisDiagnostic classificationMultiple imputationReliabilityPsicologíaCognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made availableOpen Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work has been funded by the Community of Madrid through the Pluriannual Agreement with the Universidad de Universidad Autónoma de Madrid in its Programa de Estímulo a la Investigación de Jóvenes Doctores (Reference SI3/ PJI/2021-00258), and by the Spanish Ministry of Science and Innovation (FPI BES-2016-077814)SpringerDepartamento de Psicología Social y MetodologíaFacultad de Psicología20222022-09-20research articlehttp://purl.org/coar/resource_type/c_2df8fbb1VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/10486/704451https://dx.doi.org/10.3758/s13428-022-01967-5reponame:Biblos-e Archivo. Repositorio Institucional de la UAMinstname:Universidad Autónoma de MadridInglésengopen accesshttp://purl.org/coar/access_right/c_abf2info:eu-repo/semantics/openAccessoai:repositorio.uam.es:10486/7044512026-06-23T12:46:27Z
dc.title.none.fl_str_mv Improving reliability estimation in cognitive diagnosis modeling
title Improving reliability estimation in cognitive diagnosis modeling
spellingShingle Improving reliability estimation in cognitive diagnosis modeling
Schames Kreitchmann, Rodrigo
Classification accuracy
Cognitive diagnosis
Diagnostic classification
Multiple imputation
Reliability
Psicología
title_short Improving reliability estimation in cognitive diagnosis modeling
title_full Improving reliability estimation in cognitive diagnosis modeling
title_fullStr Improving reliability estimation in cognitive diagnosis modeling
title_full_unstemmed Improving reliability estimation in cognitive diagnosis modeling
title_sort Improving reliability estimation in cognitive diagnosis modeling
dc.creator.none.fl_str_mv Schames Kreitchmann, Rodrigo
de la Torre, Jimmy
Sorrel Luján, Miguel Ángel
Najera Álvarez, Pablo
Abad García, Francisco José
author Schames Kreitchmann, Rodrigo
author_facet Schames Kreitchmann, Rodrigo
de la Torre, Jimmy
Sorrel Luján, Miguel Ángel
Najera Álvarez, Pablo
Abad García, Francisco José
author_role author
author2 de la Torre, Jimmy
Sorrel Luján, Miguel Ángel
Najera Álvarez, Pablo
Abad García, Francisco José
author2_role author
author
author
author
dc.contributor.none.fl_str_mv Departamento de Psicología Social y Metodología
Facultad de Psicología
dc.subject.none.fl_str_mv Classification accuracy
Cognitive diagnosis
Diagnostic classification
Multiple imputation
Reliability
Psicología
topic Classification accuracy
Cognitive diagnosis
Diagnostic classification
Multiple imputation
Reliability
Psicología
description Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available
publishDate 2022
dc.date.none.fl_str_mv 2022
2022-09-20
dc.type.none.fl_str_mv research article
http://purl.org/coar/resource_type/c_2df8fbb1
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv http://hdl.handle.net/10486/704451
https://dx.doi.org/10.3758/s13428-022-01967-5
url http://hdl.handle.net/10486/704451
https://dx.doi.org/10.3758/s13428-022-01967-5
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Springer
publisher.none.fl_str_mv Springer
dc.source.none.fl_str_mv reponame:Biblos-e Archivo. Repositorio Institucional de la UAM
instname:Universidad Autónoma de Madrid
instname_str Universidad Autónoma de Madrid
reponame_str Biblos-e Archivo. Repositorio Institucional de la UAM
collection Biblos-e Archivo. Repositorio Institucional de la UAM
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869406701804847104
score 15,301603

Similares en LA Referencia