Applying generalizability theory in language testing: Comparing nested and crossed scoring designs in the assessment of speaking skills

Comparing nested and crossed scoring designs in the assessment of speaking skills



Scoring language learners’ speaking skills is open to a number of measurement errors since raters’ personal judgements could involve in the process. Different grading designs in which raters score a student’s whole speaking skills or a specific dimension of the speaking performance could be settled to control and minimize the amount of the error in grading foreign language speaking skills. Therefore, the present study aims to compare G and Phi coefficients gained from the scores of a full factorial (fully crossed) model versus a nested model where rubric components are nested in graders. Four experienced raters and 116 intermediate level language learners studying at a Turkish state university’s language school voluntarily participated in this exploratory study. Findings revealed that the G and Phi values obtained with a full factorial grading model were higher. In addition, checking the variance components according to the source of variation, the variance associated with the student's main effect of the full factorial pattern was higher, while the variance value of the residual effect was lower. These findings reveal that full factorial designs could generate more reliable results in speaking exams, thus, it is recommended for language schools to implement the full factorial design in speaking exams when practical conditions such as enough time or sufficient number of graders are available.

Author Biographies

Murat Polat, Anadolu University

Murat Polat holds a Ph.D. at Osmangazi University, Institute of Educational Sciences, Research Methods and Statistics Program. Currently he is working as a language instructor at Anadolu University, School of Foreign Languages. His research interests include language assessment, alternative assessment methods, and educational statistics.

Nihan Sölpük Turhan, Fatih Sultan Mehmet Vakıf University

Koç Üniversitesi, Fen ve Edebiyat Fakültesi Matematik Bölümü mezunu olan Nihan Sölpük Turhan yüksek lisansını Sosyal Bilimler alanında İstanbul Bilgi Üniversitesinde ve doktora lisansı Eğitimde Araştırma Yöntemleri ve İstatistik alanında Eskişehir Osmangazi Üniversitesinde tamamlamıştır. Fatih Sultan Mehmet Vakıf Üniversitesi Eğitim Bilimleri Fakültesinde öğretim üyesi olup, Eğitimde Ölçme ve Değerlendirme alanında çalışmaktadır.