The reliability and sources of error of using rubrics-based assessment for student projects

Rubrics are widely used in higher education to assess performance in project-based learning environments. To date, the sources of error that may affect their reliability have not been studied in depth. Using generalisability theory as its starting-point, this article analyses the influence of the as...

Descripción completa

Detalles Bibliográficos
Autores: Menéndez Varela, José Luis, Gregori Giralt, Eva
Tipo de recurso: artículo
Estado:Versión aceptada para publicación
Fecha de publicación:2018
País:España
Institución:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
Repositorio:Recercat. Dipósit de la Recerca de Catalunya
OAI Identifier:oai:recercat.cat:2445/206971
Acceso en línea:https://hdl.handle.net/2445/206971
Access Level:acceso abierto
Palabra clave:Rúbriques d'avaluació
Educació superior
Scoring rubrics
Higher education
id ES_82614b43ccb9580da3df83a8213ee005
oai_identifier_str oai:recercat.cat:2445/206971
network_acronym_str ES
network_name_str España
repository_id_str
spelling The reliability and sources of error of using rubrics-based assessment for student projectsMenéndez Varela, José LuisGregori Giralt, EvaRúbriques d'avaluacióEducació superiorScoring rubricsHigher educationRubrics are widely used in higher education to assess performance in project-based learning environments. To date, the sources of error that may affect their reliability have not been studied in depth. Using generalisability theory as its starting-point, this article analyses the influence of the assessors and the criteria of the rubrics on the assessment of two service-learning projects. A sample of 365 novice students studying for three different undergraduate degrees was evaluated by eight student assessors and two teachers at three stages of assessment. Depending on the type of project and the stage of assessment, between 19.27 and 39.55% of the total variance was attributed to the quality of the projects, 0–7.49% to the main effect of the raters, and 3.44–17.3% to the main effect of the criteria. The results demonstrated that acceptable levels of reliability (≥.70) were obtained with three raters and eight criteria or four raters and nine criteria in contexts of relative or absolute decisions, respectively.Taylor & Francis2024202420182024info:eu-repo/semantics/articleinfo:eu-repo/semantics/acceptedVersion11 p.application/pdfhttps://hdl.handle.net/2445/206971Articles publicats en revistes (Història de l'Art)reponame:Recercat. Dipósit de la Recerca de Catalunyainstname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)InglésVersió postprint del document publicat a: https://doi.org/10.1080/02602938.2017.1360838Assessment & Evaluation In Higher Education, 2018, vol. 43, num.3, p. 488-499https://doi.org/10.1080/02602938.2017.1360838(c) Taylor & Francis, 2018info:eu-repo/semantics/openAccessoai:recercat.cat:2445/2069712026-05-29T05:05:01Z
dc.title.none.fl_str_mv The reliability and sources of error of using rubrics-based assessment for student projects
title The reliability and sources of error of using rubrics-based assessment for student projects
spellingShingle The reliability and sources of error of using rubrics-based assessment for student projects
Menéndez Varela, José Luis
Rúbriques d'avaluació
Educació superior
Scoring rubrics
Higher education
title_short The reliability and sources of error of using rubrics-based assessment for student projects
title_full The reliability and sources of error of using rubrics-based assessment for student projects
title_fullStr The reliability and sources of error of using rubrics-based assessment for student projects
title_full_unstemmed The reliability and sources of error of using rubrics-based assessment for student projects
title_sort The reliability and sources of error of using rubrics-based assessment for student projects
dc.creator.none.fl_str_mv Menéndez Varela, José Luis
Gregori Giralt, Eva
author Menéndez Varela, José Luis
author_facet Menéndez Varela, José Luis
Gregori Giralt, Eva
author_role author
author2 Gregori Giralt, Eva
author2_role author
dc.subject.none.fl_str_mv Rúbriques d'avaluació
Educació superior
Scoring rubrics
Higher education
topic Rúbriques d'avaluació
Educació superior
Scoring rubrics
Higher education
description Rubrics are widely used in higher education to assess performance in project-based learning environments. To date, the sources of error that may affect their reliability have not been studied in depth. Using generalisability theory as its starting-point, this article analyses the influence of the assessors and the criteria of the rubrics on the assessment of two service-learning projects. A sample of 365 novice students studying for three different undergraduate degrees was evaluated by eight student assessors and two teachers at three stages of assessment. Depending on the type of project and the stage of assessment, between 19.27 and 39.55% of the total variance was attributed to the quality of the projects, 0–7.49% to the main effect of the raters, and 3.44–17.3% to the main effect of the criteria. The results demonstrated that acceptable levels of reliability (≥.70) were obtained with three raters and eight criteria or four raters and nine criteria in contexts of relative or absolute decisions, respectively.
publishDate 2018
dc.date.none.fl_str_mv 2018
2024
2024
2024
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/acceptedVersion
format article
status_str acceptedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/2445/206971
url https://hdl.handle.net/2445/206971
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Versió postprint del document publicat a: https://doi.org/10.1080/02602938.2017.1360838
Assessment & Evaluation In Higher Education, 2018, vol. 43, num.3, p. 488-499
https://doi.org/10.1080/02602938.2017.1360838
dc.rights.none.fl_str_mv (c) Taylor & Francis, 2018
info:eu-repo/semantics/openAccess
rights_invalid_str_mv (c) Taylor & Francis, 2018
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv 11 p.
application/pdf
dc.publisher.none.fl_str_mv Taylor & Francis
publisher.none.fl_str_mv Taylor & Francis
dc.source.none.fl_str_mv Articles publicats en revistes (Història de l'Art)
reponame:Recercat. Dipósit de la Recerca de Catalunya
instname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
instname_str Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
reponame_str Recercat. Dipósit de la Recerca de Catalunya
collection Recercat. Dipósit de la Recerca de Catalunya
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869412042288398336
score 15,81155

Similares en LA Referencia