Evaluating ChatGPT's Reliability in Second Language Acquisition (SLA): Insights on Language Skills and Technology's Role

Abstract

Artificial Intelligence (AI) has become a transformative tool across various fields, including education, offering innovative opportunities for second language (L2) learning and teaching. The potential of AI-driven tools like ChatGPT in Second Language Acquisition (SLA) remains an area of emerging interest, with implications for enhancing pedagogical approaches and providing scalable, interactive support for learners and educators. This study evaluates the reliability and quality of ChatGPT-4.0’s responses to 48 key questions related to SLA, encompassing themes such as technology's role and language skills (reading, writing, listening, and speaking). The questions were rated by five expert linguists using a Likert scale to assess accuracy, and relevance. Statistical analyses, including the Kruskal-Wallis H test (p < 0.05) and post-hoc pairwise comparisons, revealed significant differences among evaluators (p < 0.001), with most responses rated as good (26.25%) or very good (55.41%). Despite these positive outcomes, Fleiss’s Kappa coefficient (0.006) highlighted low inter-rater agreement, signaling a need for standardized evaluation frameworks. While ChatGPT shows promise in addressing diverse SLA topics, its limitations—such as occasional inaccuracies—underscore the necessity for ongoing research to optimize its utility in educational contexts. This study contributes to SLA pedagogy by demonstrating the potential and boundaries of integrating AI into language learning, advocating for cautious and evidence-based adoption.