LLM 기반 맞춤형 자동문항생성기의 활용가능성 탐색

Exploring the potential of LLM-based customized test item generators: Focusing on Poe AI as a chatbot builder

The study aims to enhance prompt literacy among pre-service teachers by developing an automated test item generator using Poe AI, a free chatbot builder platform based on Large Language Models (LLMs). An experiment was conducted with 84 pre-service teachers enrolled at a university of education in 2024. The experiment lasted two hours, with one hour dedicated to training and practice in chatbot creation, followed by one hour dedicated to creating an item generator and participating in a survey. The pre-service teachers successfully created curriculum-based item generators. However, due to time constraints, they could only focus on one to four item types. The survey revealed that 90.5% of the pre-service teachers had no prior experience in building a chatbot before the experiment. As a result, 65.5% found the process somewhat difficult, and 25.0% found writing prompts very difficult. Despite facing challenges, most feedback on the quality of the items generated by their programs was positive. Only 7.1% reported that the items were immediately usable, but 46.4% stated that the items could be used with minor modifications. These findings highlight the importance of providing prompt literacy training for pre-service teachers. In an open-ended questionnaire, the participants identified prompt literacy as a critical future core competency in the context of AI-human collaboration.