Rubric Development and Validation for Assessing Tasks' Solving via AI Chatbots

Authors

DOI:

https://doi.org/10.34190/ejel.22.6.3292

Keywords:

Rubric development, AI chatbot, Validity, Tasks assessment, Educational technology

Abstract

This research aimed to develop and validate a rubric to assess Artificial Intelligence (AI) chatbots' effectiveness in accomplishing tasks, particularly within educational contexts. Given the rapidly growing integration of AI in various sectors, including education, a systematic and robust tool for evaluating AI chatbot performance is essential. This investigation involved a rigorous process including expert involvement to ensure content validity, as well as the application of statistical tests for assessing internal consistency and reliability. Factor analysis also revealed two significant domains, "Quality of Content" and "Quality of Expression", which further enhanced the construct validity of the evaluation scale. The results from this investigation robustly affirm the reliability and validity of the developed rubric, thus marking a significant advancement in the sphere of AI chatbot performance evaluation within educational contexts. Nonetheless, the study simultaneously emphasizes the requirement for additional validation research, specifically those entailing a variety of tasks and diverse AI chatbots, to further corroborate these findings. The ramifications of this research are profound, offering both researchers and practitioners engaged in chatbot development and evaluation a comprehensive and validated framework for the assessment of chatbot performance.

Author Biographies

Mohammad Hmoud, Faculty of Educational Sciences, An-Najah National University, Nablus, Palestine

Mohammad Hmoud is a Ph.D. candidate at An-Najah National University, specializes in integrating emerging technologies like AI and big data into education. Working as an education supervisor in Jerusalem's Ministry of Education, he lectures on educational technology and programming. His research seeks innovative methods to enhance teaching and learning experiences.

Hadeel Swaity, Faculty of Educational Sciences, An-Najah National University, Nablus, Palestine

Hadeel Swaity is a Ph.D. candidate at An-Najah National University, has degrees in Educational Supervision and Occupational Therapy. Working at Matya, East Jerusalem, she aids those with diverse needs. Her research advances inclusive education, emphasizing technology, self-regulated learning, and primary writing skills.

Eman Anjass, Faculty of Educational Sciences, University of Granada, Spain

Eman Anjass is an experienced science educator with 12 years in the field, garnered acclaim as the "Outstanding Teacher in Palestine" in 2019. Currently pursuing a Ph.D. at Granada University in Spain, her research revolves around the integration of technology in science education.

Eva María Aguaded-Ramírez, Faculty of Educational Sciences, University of Granada, Spain

Eva María Aguaded-Ramírez is a full Professor at University of Granada, Spain, she has a diverse research background spanning education, emotional intelligence, and AI chatbots. Her recent work focuses on teacher training, emotional education, and personal learning environments, particularly for marginalized groups like unaccompanied immigrant and refugee children. She has published extensively in various academic journals and books, addressing topics such as critical thinking, medical education, emotional intelligence, and attitudes toward refugee children.

Downloads

Published

17 May 2024

Issue

Vol. 22 No. 6 (2024): Special Issue: Artificial Intelligence (AI) in Education Part 2

Section

Articles

License

Copyright (c) 2024 Mohammad Hmoud, Hadeel Swaity, Eman Anjass, Eva María Aguaded-Ramírez

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Open Access Publishing

The Electronic Journal of e-Learning operates an Open Access Policy. This means that users can read, download, copy, distribute, print, search, or link to the full texts of articles, crawl them for indexing, pass them as data to software, or use them for any other lawful purpose, without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself. The only constraint on reproduction and distribution, and the only role for copyright in this domain, is that authors control  the integrity of their work, which should be properly acknowledged and cited.