Examining the Sources of Variability in Writing Test Scores

In the present study, in order to better understand the complex nature of writing performance assessment, factors affecting writing test scores were investigated in a systematic analysis in which the effects of these factors could be disentangled. Specifically, the primary purpose of this study was to investigate the extent to which the students’ performance on a writing test was affected by their underlying writing ability as opposed to the test method factors such as specific tasks and raters used to elicit and score test performance. For the purpose of this study, generalizability theory analysis (Brennan, 2001), which enables us to estimate the relative effects of various factors affecting test scores, was used. The results showed that the students’ scores on the writing test were significantly affected by the method effects associated with the writing tasks used. Given the instability of students’ performances across tasks observed in the present study, the way forward seems to recognize that language testers need to take into account both a construct-based and a task-based approach to test design and score interpretation.