The purpose of this study is to verify whether ChatGPT can be used as a reliable automated scoring tool in English writing courses. ChatGPT’s reliability has been somewhat verified through large-scale assessments, while its potential as a reliable evaluation tool for small classes in university education settings remains unproven. In this study, ChatGPT-3.5 was used to repeatedly assess 30 English writing tasks based on the IELTS Writing Scale, and a repeated measures analysis of variance was used to statistically verify whether ChatGPT provides consistent responses. The results are summarized as follows. First, ChatGPT was found to be a highly reliable scoring tool with very high internal consistency and high inter-component correlations. Second, the reliability of ChatGPT depends on the scoring criteria. A detailed rubric increases the reliability of scoring. Third, the reliability of ChatGPT depends on the scoring scale. Given constructs only, ChatGPT can rate on a holistic scoring scale, but given constructs and a detailed rubric it can rate on holistic or analytical scales. While this study focused on ChatGPT’s reliability, if avenues for establishing validity are explored, generative AI like ChatGPT could become widely applicable as scoring instruments that contribute to achieving educational goals in English composition education.
.
(0)
(0)