Development and Application of AI Agents for Elementary English Writing Assessment: Focusing on a Comparison with ChatGPT

This study developed and examined AIEEWA, a specialized AI agent system for elementary English writing assessment, by comparing its utility against the general-purpose model ChatGPT. AIEEWA comprises two agents: (a) an automatic question generation (AQG) agent that generates curriculum-aligned questions and rubrics and (b) an automated answer scoring (AAS) agent that scores answers and generates feedback. The system was developed by leveraging retrieval-augmented generation (RAG) and self-evaluation mechanisms (Self-RAG, LLM-as-a-Judge) with GPT-4o. Its performance was evaluated using questions generated by the agent and writing samples collected from Korean students in the 5th and 6th grades. Results show that AIEEWA’s AQG agent generated questions with a higher average content validity than those generated by ChatGPT, although expert review remains indispensable. The AAS agent also demonstrated significantly higher intra-rater scoring reliability and feedback consistency than ChatGPT. In addition, although AIEEWA’s AAS agent achieved better inter-rater agreement with teacher consensus scores than ChatGPT, the agreement level was still moderate, underscoring the difficulty of fully replicating nuanced human judgment. These findings suggest that purpose-built AI agents such as AIEEWA can substantially enhance assessment efficiency and consistency compared with ChatGPT. At the current stage, however, they are better suited to assisting teachers rather than replacing them. The code is available at https://colab.research.google.com/drive/1Iy4Fxcj-8mY3wgTDj IGHhy6_M0xCQ4Vf?usp=sharing