Applicability of an EBB Scale to English Speaking Performance Assessment in the Korean Secondary School

This study investigates not only some of the structural issues of an EBB rating scale but also rater reliability in terms of its stability in the classroom context. A group of three teachers developed two EBB scales: a four-level EBB for linguistic accuracy and a six-level EBB for communicative effectiveness. The findings on structural traits of an EBB scale are: ⅰ) the binary question format of an EBB scale does not prevent confusion in perceiving some subjective linguistic factors; ⅱ) the top questions exert too influential power on subsequent questions or sub-groups; ⅲ) an EBB scale is so inflexible that it cannot reflect linguistic traits that do not appear in a particular level; ⅳ) the limited number of questions cannot cover some unique linguistic traits of a few examinees. However, a FACETS analysis shows that an EBB scale is effective in ensuring high rater reliability. Raters show similar rater severity, enough intra-rater reliability without any misfit or overfit, and high ICC inter-rater reliability with .83 for linguistic accuracy and with .89 for communicative effectiveness.