Applying generalizability theory in language testing: Comparing nested and crossed scoring designs in the assessment of speaking skills
Comparing nested and crossed scoring designs in the assessment of speaking skills
Scoring language learners’ speaking skills is open to a number of measurement errors since raters’ personal judgements could involve in the process. Different grading designs in which raters score a student’s whole speaking skills or a specific dimension of the speaking performance could be settled to control and minimize the amount of the error in grading foreign language speaking skills. Therefore, the present study aims to compare G and Phi coefficients gained from the scores of a full factorial (fully crossed) model versus a nested model where rubric components are nested in graders. Four experienced raters and 116 intermediate level language learners studying at a Turkish state university’s language school voluntarily participated in this exploratory study. Findings revealed that the G and Phi values obtained with a full factorial grading model were higher. In addition, checking the variance components according to the source of variation, the variance associated with the student's main effect of the full factorial pattern was higher, while the variance value of the residual effect was lower. These findings reveal that full factorial designs could generate more reliable results in speaking exams, thus, it is recommended for language schools to implement the full factorial design in speaking exams when practical conditions such as enough time or sufficient number of graders are available.
Copyright (c) 2021 International Journal of Curriculum and Instruction
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Submission of an article implies that the work described has not been published previously (except in the form of an abstract or as part of a published lecture or academic thesis), that it is not under consideration for publication elsewhere, that its publication is approved by all authors and tacitly or explicitly by the responsible authorities where the work was carried out, and that, if accepted, will not be published elsewhere in the same form, in English or in any other language, without the written consent of the Publisher. The Editors reserve the right to edit or otherwise alter all contributions, but authors will receive proofs for approval before publication.
Copyrights for articles published in International Journal of Curriculum and Instruction are retained by the authors, with first publication rights granted to the journal. The journal/publisher is not responsible for subsequent uses of the work. It is the author's responsibility to bring an infringement action if so desired by the author.