Score Reliability of a Test Composed of Passage-Based Testlets - A Generalizability Theory Perspective

Score Reliability of a Test Composed of Passage-Based Testlets

  The purpose of a study reported in this paper is to investigate the impact of local item dependence (LID) in passage-based testlets on the score reliability of an EFL (English as a Foreign Language) reading comprehension test from the perspective of generalizability (G) theory. Definitions and causes of LID in passage-based testlets are reviewed within the frameworks of classical test theory (CTT) and item response theory (IRT) along with 3 reliability estimation procedures for testlet-based tests. Scored item responses of 1,072 Korean high school students to a 40-item, 10-passage EFL reading comprehension test were analyzed with simulated data generated for a no LID situation for a 40-item test. It was found that item alpha (α<SUB>I</SUB>) coefficients (p x I) were significantly higher than both passage alpha (α<SUB>p</SUB>) coefficients (p x T) and generalizability (Eρ²) coefficients [p x (I:H)] in the real data, whereas all of these 3 coefficients were very close for different test lengths in the simulated data. Subsequent analyses of fixed test length scenarios have also revealed that score reliability could be maximized by increasing the number of passages rather than the number of items nested within passages. The findings of the study are also discussed in terms of the impact of LID on score reliability and its implications for test development.