감성분석에 기반한 수능 영어영역 ‘심경·분위기 파악’ 문항의 타당도 분석 연구

A validity study on the item type of ‘identifying the emotional state or mood’ in the English section of CSAT based on sentiment analysis

The present study attempts to validate the item type of ‘identifying the emotional state or mood’ in the English section of the CSAT by using sentiment analysis. To this end, the agreement rate between the analysis results based on two sentiment analysis tools, ‘Tone analyzer’ and ‘KH-Coder,’ and the correct answers of the 60 target items from the CSATs 1994-2020 was examined. The results of this study show that ‘Tone Analyzer’ has an agreement rate of about 67%, while the agreement rate of ‘KH-Coder’ is about 57%. That is, the result of ‘Tone Analyzer’ adapting a machine learning method is more accurate than that of ‘KH-Coder’, which is based on a sentiment dictionary method. Additionally, sentiment words, possible explicit cues to infer a correct answer, appeared in most of the 60 items. However, in the case of an item where any sentiment words were not directly presented, both sentiment analysis tools mostly failed to find the correct answer. This means that the target item type needs to be developed as an item requiring contextual information to infer in order to reinforce its construct validity. In conclusion, some pedagogical implications and suggestions for future studies are addressed.