Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel

Little is known about the measurement quality of questions in web surveys, even if, this information is crucial to design better questionnaires and to correct for measurement errors in substantive analyses. This paper aims to cover this gap by answering the following four objectives. The first objec...

ver descrição completa

Detalhes bibliográficos
Autores: DeCastellarnau, Anna, Revilla, Melanie
Formato: artículo
Estado:Versión publicada
Fecha de publicación:2017
País:España
Recursos:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
Repositorio:Recercat. Dipósit de la Recerca de Catalunya
OAI Identifier:oai:recercat.cat:10230/69708
Acesso em linha:http://hdl.handle.net/10230/69708
http://dx.doi.org/10.18148/srm/2017.v11i4.7226
Access Level:acceso abierto
Palavra-chave:Measurement quality
Multitrait-multimethod
SQP
Web surveys
Norwegian citizen panel
id ES_dac400c7feed28a30b7d29f8a701a46e
oai_identifier_str oai:recercat.cat:10230/69708
network_acronym_str ES
network_name_str España
repository_id_str
spelling Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen PanelDeCastellarnau, AnnaRevilla, MelanieMeasurement qualityMultitrait-multimethodSQPWeb surveysNorwegian citizen panelLittle is known about the measurement quality of questions in web surveys, even if, this information is crucial to design better questionnaires and to correct for measurement errors in substantive analyses. This paper aims to cover this gap by answering the following four objectives. The first objective, is to evaluate the measurement quality of a set of survey questions from two Multitrait-Multimethod (MTMM) experiments implemented in the 5th wave of the Norwegian Citizen Panel ; one of the few probability-based online panels existing at this day. Each experiment is designed to evaluate three different formulations of the response scale for the topics: political satisfaction and trust in the institutions. The second objective is to predict the measurement quality of these questions by its design characteristics, using the software Survey Quality Predictor (SQP). The third, is to compare the quality of the different formulations of the response scale used. The fourth, is to compare both the MTMM and the SQP approaches to assess whether both can lead to similar results when evaluating web survey questions. Overall, measurements’ quality is quite high (between 0.60 and 0.89), and similar between the estimates obtained from the MTMM experiments and the SQP predictions. On the one hand, we conclude that when comparing the different scales, the horizontal 11-point scale with 2 fixed reference points and ordered from negative to positive, usually, provides the highest quality. On the other hand, we conclude that SQP can provide as accurate quality predictions as MTMM can estimate the quality for web survey questions. Given that each approach has its advantages and limitations, when possible we recommend using both to correct for measurement errors, as kind of sensitivity analysis.European Survey Research Association202520252017info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfapplication/pdfhttp://hdl.handle.net/10230/69708http://dx.doi.org/10.18148/srm/2017.v11i4.7226reponame:Recercat. Dipósit de la Recerca de Catalunyainstname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)InglésSurvey Research Methods. 2017 Dec 13;11(4):415-33Copyright for articles published in this journal is retained by the authors, with first publication rights granted to the journal. By virtue of their appearance in this open access journal, users can use, reuse and build upon the material published in the journal but only for non-commercial purposes and with proper attribution.http://creativecommons.org/licenses/by-nc/4.0/deed.eninfo:eu-repo/semantics/openAccessoai:recercat.cat:10230/697082026-05-29T05:05:01Z
dc.title.none.fl_str_mv Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
title Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
spellingShingle Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
DeCastellarnau, Anna
Measurement quality
Multitrait-multimethod
SQP
Web surveys
Norwegian citizen panel
title_short Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
title_full Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
title_fullStr Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
title_full_unstemmed Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
title_sort Two approaches to evaluate measurement quality in online surveys: an application using the Norwegian Citizen Panel
dc.creator.none.fl_str_mv DeCastellarnau, Anna
Revilla, Melanie
author DeCastellarnau, Anna
author_facet DeCastellarnau, Anna
Revilla, Melanie
author_role author
author2 Revilla, Melanie
author2_role author
dc.subject.none.fl_str_mv Measurement quality
Multitrait-multimethod
SQP
Web surveys
Norwegian citizen panel
topic Measurement quality
Multitrait-multimethod
SQP
Web surveys
Norwegian citizen panel
description Little is known about the measurement quality of questions in web surveys, even if, this information is crucial to design better questionnaires and to correct for measurement errors in substantive analyses. This paper aims to cover this gap by answering the following four objectives. The first objective, is to evaluate the measurement quality of a set of survey questions from two Multitrait-Multimethod (MTMM) experiments implemented in the 5th wave of the Norwegian Citizen Panel ; one of the few probability-based online panels existing at this day. Each experiment is designed to evaluate three different formulations of the response scale for the topics: political satisfaction and trust in the institutions. The second objective is to predict the measurement quality of these questions by its design characteristics, using the software Survey Quality Predictor (SQP). The third, is to compare the quality of the different formulations of the response scale used. The fourth, is to compare both the MTMM and the SQP approaches to assess whether both can lead to similar results when evaluating web survey questions. Overall, measurements’ quality is quite high (between 0.60 and 0.89), and similar between the estimates obtained from the MTMM experiments and the SQP predictions. On the one hand, we conclude that when comparing the different scales, the horizontal 11-point scale with 2 fixed reference points and ordered from negative to positive, usually, provides the highest quality. On the other hand, we conclude that SQP can provide as accurate quality predictions as MTMM can estimate the quality for web survey questions. Given that each approach has its advantages and limitations, when possible we recommend using both to correct for measurement errors, as kind of sensitivity analysis.
publishDate 2017
dc.date.none.fl_str_mv 2017
2025
2025
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/10230/69708
http://dx.doi.org/10.18148/srm/2017.v11i4.7226
url http://hdl.handle.net/10230/69708
http://dx.doi.org/10.18148/srm/2017.v11i4.7226
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Survey Research Methods. 2017 Dec 13;11(4):415-33
dc.rights.none.fl_str_mv http://creativecommons.org/licenses/by-nc/4.0/deed.en
info:eu-repo/semantics/openAccess
rights_invalid_str_mv http://creativecommons.org/licenses/by-nc/4.0/deed.en
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv European Survey Research Association
publisher.none.fl_str_mv European Survey Research Association
dc.source.none.fl_str_mv reponame:Recercat. Dipósit de la Recerca de Catalunya
instname:Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
instname_str Varias* (Consorci de Biblioteques Universitáries de Catalunya, Centre de Serveis Científics i Acadèmics de Catalunya)
reponame_str Recercat. Dipósit de la Recerca de Catalunya
collection Recercat. Dipósit de la Recerca de Catalunya
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869421610359848960
score 15.812429

Registos relacionados