Factors Associated with Improvement of Mathematical Definitions in AI-Mediated Conversations

Large language models (LLMs) are increasingly integrated into educational settings, and mathematics educators face new opportunities to explore how AI-mediated dialogue can support students' conceptual understanding. This proof-of-concept study investigated features of LLM-mediated conversations that impacted students' refinements of mathematics definitions. The authors generated 646 definition refinements for two geometric objects (ray and quadrilateral) using structured feedback prompts in 216 simulated conversations using three LLMs (ChatGPT-4, Copilot, and Gemini). The study addressed two research questions: (1) Which features of a simulated LLM definition refinement conversation are associated with producing an improved definition? and (2) In simulated LLM definition refinement conversations using a questioning feedback strategy, how is question type (funneling vs. focusing) associated with the likelihood of an improved definition? Logistic regression analyses revealed mathematical object, initial definition, refinement number, feedback structure, and LLM choice influenced the odds of definition improvement. Funneling questions were associated with higher odds of improvement than focusing questions—though this finding is interpreted cautiously given the simulation-based design. Definition improvements were more likely for ray than quadrilateral, and early refinements were more likely to succeed than later ones. ChatGPT outperformed Gemini, particularly for quadrilaterals. Results suggest LLMs can support productive definition revision, but the nature of the simulation methodology limits what conclusions can be drawn related to student learning. Findings are best understood as evidence of task design viability. Future research should examine actual students' interactions with LLMs, exploring how feedback structure, prompt design, and LLM behavior influence mathematical understanding during formative assessment LLM conversations.