An Investigation into the Variability of a Writing Test Using Generalizability Theory

An Investigation into the Variability of a Writing Test Using Generalizability Theory

  Because performance assessment such as a composition test introduces a range of factors that may influence the chances of success for a candidate on the test, those in charge of monitoring quality control for performance assessment programs need to gather information that will help them determine whether all aspects of the programs are working as intended. In the present study, generalizability theory (Brennan, 1992) was employed to examine the relative effects of various sources of variability on students" performance on an essay writing test and also to investigate the reliability of the assigned scores. The results showed that due to the largest effect associated with the students" writing ability and negligible effects associated with facets of measurement such as scoring criteria and ratings, the generalizability coefficient estimated for the writing test was high, suggesting that the test is a reliable measure of what it purports to measure, the students" writing ability.