학문 목적 한국어 쓰기 능력에 대한 분석적 평가의 일반화가능도 검증

Analysis of Error Sources and Estimation of Reliability in a Analytic Evaluation on Writing Ability for Academic Purpose Korean by Applying Generalizability Theory

The primary aim of this study is: 1) to determine error sources and the effects of each error source in an analytic evaluation on APK(Academic Purpose Korean) writing ability; 2) to estimate the generalizability coefficient; and 3) to suggest optimal measuring conditions for an analytic evaluation on APK writing ability based on the generalizability theory. For the design of the G study(Generalizability Study), a three-facet crossed model(p x r x i) was used with two academic writing task(a test with 2 hour limit and a report with 1 week term), 9 raters, and 10 analytic scoring items. The D study(Decision Study) design was used in an identical manner as the G study. The result of the G study for estimating error sources in the analytic evaluation on APK writing ability indicates that error sources for raters have a more significant impact than error sources related to analytic scoring items associated with the score of the writing test. As for the generalizability coefficients in the D study, increasing the number of the raters had more significant impact than increasing the number of assessment items in the analytic evaluation on APK writing ability. However even the ratio in the error dispersion of the raters is not small, the ratio in the error dispersion of the items is 0. It means that these 10 analytic scoring items guided a coherent response from 9 raters. Therefore, it seems useful to train the raters with these 10 analytic scoring items.