Investigating ChatGPT’s phonology problem-solving abilities through reasoning with varying custom instructions

In this paper, we investigate ChatGPT’s ability to solve phonological problems with varying custom instructions. It is known that ChatGPT has difficulty solving unfamiliar novel problems, and phonology provides a good test case for evaluating its reasoning abilities when faced with such challenges. We designed phonological problems using data or rules that are not observed in the phonology literature. We varied model versions (GPT-4 and GPT-4o) and custom instructions by varying the levels of knowledge: no custom instructions, beginner(‘Student’), and expert (‘Professor’) levels. Four novel questions were created by modifying the following phonological data: (1) Korean fricative palatalization, (2)Spanish intervocalic lenition, (3) Rule ordering in Canadian English, and (4) English syllabification. ChatGPT exhibits some level of reasoning capabilities, with test scores ranging from 27% to 75%. GPT-4o performed better than GPT-4. However, it made errors when dealing with novel patterns or rules. The effects of varying custom instructions were present but less clear and inconsistent. Our results suggest that,depending on the model version and customization, GPT can learn phonologically unnatural processes through reasoning.