Investigating variability in tasks, analytic scoring criteria, and rater groups on L2 writing in English

The purpose of the study is to investigate how the judgment of raters is biased toward certain rating criteria or task types in assessing Korean university students’ L2 writing. In addition, the study aims to examine whether or not non-native rater group and native rater group share the same judgment of students’ written products. Six raters scored 258 compositions written by Korean university students on two topics. The written products were rated using an analytic scoring scale with content, organization, language in use, grammar, and mechanics. For the first part, the FACETS analysis revealed that all six raters showed self-consistent scoring patterns across the rating categories in the two tasks. However, each rater was biased towards certain type of tasks and rating categories. For the second part of the study, the results of ANOVA and MANOVA tests indicated that there were significant mean differences between the two groups across the rating categories in both tasks except content in Task 2. It is shown that the non-native rater group is harsher in their rating than the native rater group regardless of the rating categories.