고등학생 영어학습자의 말하기 평가에서 채점자 훈련의 효과 및 채점자 인식 분석

An analysis of rater perceptions and the effectiveness of the rater training in speaking assessment

This study aims to examine the effectiveness of rater training as to the rater reliability in evaluating speaking skill, and to analyze how teachers’ perceptions changed as the training proceeded. For the research, four English teachers in Korean high school rated 15 students four times, once a week, and were interviewed after each rating period. The scoring data was analyzed through the Many-Facet Rash Measurement (MFRM) and the interviews were also analyzed. The results are as follows. First, both the inter- and intra-rater reliability increased. Although four raters were all leniently biased from the first session, the difference of the degree of leniency was closely narrowed down. Also, misfit aspect observed from one of the raters disappeared as the biases incurred by the lack of understanding of the scoring criteria reduced. Second, contrary to their previous concerns about the result of the training, teachers agreed on the need for the training and its effectiveness. The interview revealed that the training encouraged teachers to have confidence in their expertise, eventually raising reliability in their rating. The study demonstrates how rater training can establish rater reliability and most importantly implies that it can be successfully settled in Korean high school.