초등영어 쓰기 평가에서 생성형 인공지능을 활용한 자동채점 가능성 탐색

Exploring the feasibility of automated scoring using generative AI in elementary English writing

This study explores the feasibility of automated scoring using generative AI (ChatGPT, Claude, and WorkAI) in an elementary English writing assessment. Essays from 87 elementary school students for English essay contest were analyzed using intraclass correlation coefficients (ICC) to examine agreement levels between human raters, among generative AI models, and between human raters and generative AI models. The results are as follows: (1) Agreement between human raters was generally high, but differences were observed in Item 2, particularly in the 'organization' criterion. (2) Agreement among generative AI models was good overall, with lenient scoring resulting in higher agreement for 'content,' 'organization,' and 'total score,' while stricter scoring worked better for 'accuracy' and 'range & mechanics.' (3) Agreement between human raters and generative AI models ranged from moderate to high. Stricter scoring produced higher agreement for Item 1 across all criteria, while lenient scoring worked better for 'content' and 'organization' in Item 2. These findings suggest that generative AI has potential as an automated scoring tool. Adjusting rater tendencies based on tasks and criteria allows generative AI to reflect human scoring patterns, supporting its use as a complementary tool in educational assessment.