AI-Automated EFL Writing Assessment Using Standardized Rubrics

This study investigates the reliability of AI-automated writing assessment in university-level EFL composition classes. The research compares human assessment and AI-generated assessment of 32 paragraphs written by Korean university students, using a standardized rubric adapted from the University of Michigan Writing Center. The study utilized Claude 3.5 Sonnet, accessed via API calls, to conduct multiple assessment iterations under controlled conditions. The assessment criteria included Content and Ideas, Organization and Structure, Language Use and Vocabulary, and Grammar and Mechanics, each scored on a five-point scale. Statistical analysis revealed high consistency in AI assessment (r=0.979). While correlation between human raters was moderate (r = 0.507), the AI system demonstrated reliable assessment patterns that aligned significantly with human scoring trends. The findings suggest that AI-automated assessment systems can provide reliable evaluation of student writing, though with some notable differences from human assessment patterns. This study contributes to the emerging field of AI-assisted writing assessment and provides a foundation for developing more sophisticated automated feedback systems for EFL writing instruction.