Application of Confirmatory Factor Analysis to the Validity Study of a Performance Assessment

This study investigated construct validity and factorial invariance of a performance assessment of reading comprehension and writing proficiency, through a multitrait-multimethod structure (MTMM), using the confirmatory analysis technique. First, interrater reliability was examined for each measured variable using three different generalizability coefficients. Although all of the measures were found to be highly reliable, exploratory factor analysis indicated that trait and method effects were confounded in the measured variables. Consequently, confirmatory factor analysis was used to disentangle multidimensionality and examine the convergent and discriminant validity of the latent variables according to the Campbell-Fiske criteria. These analyses indicated that a model with three correlated trait factors and three correlated method factors provided the best fit to the data. Finally, a factorial invariance across gender and grade was examined. While this MTMM factor structure was fitted to the data in each subgroup (fifth grade boys. fifth grade girls, sixth grade girls), the factorial invariance across gender and grade was supported only in a particular set of parameters. Methodological and practical implications of the use of confirmatory factor analysis in multitrait-multimethod analyses are also discussed for construct validation in performance assessment across different groups.