The Accuracy of AI-Based Automatic Proctoring in Online Exams
DOI:
https://doi.org/10.34190/ejel.20.4.2600Keywords:
AI-based proctoring, automatic proctoring, online exams, software accuracy, academic integrityAbstract
This study technically analyses one of the online exam supervision technologies, namely the Artificial Intelligence-based Auto Proctoring (AiAP). This technology has been heavily presented to the academic sectors around the globe. Proctoring technologies are developed to provide oversight and analysis of students’ behavior in online exams using AI, and sometimes with the supervision of human proctors to maintain academic integrity in a blended format. Manual Testing methodology was used to do a software testing on AiAP for verification of any possible incorrect red flags or detections. The study took place in a Middle Eastern university by conducting online exams for 14 different courses, with a total of 244 students. Afterward, five human proctors were assigned to verify the data obtained by the AiAP software. The results were then compared in terms of monitoring measurements: screen violation, sound of speech, different faces, multiple faces, and eyes movement detection. The proctoring decision was computed by averaging all monitoring measurements and then compared between the human proctors’ and the AiAP decisions, to ultimately set the AiAP against a benchmark (human proctoring) and hence to be viable for use. The decision represented the number of violations to the exam conditions, and the result showed a significant difference between Human Decision (average 25.95%) and AiAP Decision (average 35.61%), and the total number of incorrect decisions made by AiAP was 74 out of 244 exam attempts, concluding that AiAP needed some improvements and updates to meet the human level. The researchers provided some technical limitations, privacy concerns, and recommendations to carefully review before deploying and governing such proctoring technologies at institutional level. This paper contributes to the field of educational technology by providing an evidence-based accuracy test on an automatic proctoring software, and the results demand institutional provision to better establish an appropriate online exam experience for higher educational institutions.
References
Alessio, H.M., Malay, N., Maurer, K., Bailer, A.J. and Rubin, B., 2017. Examining the effect of proctoring on online test scores. Online Learning, 21(1), pp.146-161. http://doi.org/10.24059/olj.v21i1.885.
Anwar, N. and Kar, S., 2019. Review paper on various software testing techniques & strategies. Global Journal of Computer Science and Technology, 19(2), pp.43-49. Available at: <https://computerresearch.org/index.php/computer/article/view/1873> [Accessed 11 June 2021].
Clark, M., 2021. Students of color are getting flagged to their teachers because testing software can’t see them. [online] Available at: <https://www.theverge.com/2021/4/8/22374386/proctorio-racial-bias-issues-opencv-facial-detection-schools-tests-remote-learning> [Accessed 12 December 2021].
Dendir, S. and Maxwell, R.S., 2020. Cheating in online courses: Evidence from online proctoring. Computers in Human Behavior Reports 2. [Advance online publication]. https://doi.org/10.1016/j.chbr.2020.100033.
Google Codelabs., 2021. Using the Vision API with Python. [online] Available at: <https://codelabs.developers.google.com/codelabs/cloud-vision-api-python#0> [Accessed 10 February 2022].
Hussein, M.J., Yusuf, J., Deb, A.S., Fong, L. and Naidu, S., 2020. An evaluation of online proctoring tools. Open Praxis, 12(4), pp.509-525. https://doi.org/10.3316/informit.620366163696963.
Ismail, R., Safieddine, F. and Jaradat, A., 2019. E-university delivery model: Handling the evaluation process. Business Process Management Journal, 25(7), pp.1633-1646. https://doi.org/10.1108/BPMJ-10-2018-0281.
Ilgaz, H. and Afacan Adanır, G., 2020. Providing online exams for online learners: Does it really matter for them? Education and Information Technologies, 25(2), pp.1255-1269. https://doi.org/10.1007/s10639-019-10020-6.
Kaiiali, M., Ozkaya, A., Altun, H., Haddad, H. and Alier, M., 2016. Designing a secure exam management system (SEMS) for M-learning environments. IEEE Transactions on Learning Technologies, 9(3), pp.258-271. https://doi.org/10.1109/TLT.2016.2524570.
Kharbat, F.F. and Abu Daabes, S.A., 2021. E-proctored exams during the COVID-19 pandemic: A close understanding. Education and Information Technologies. [Advance online publication]. https://doi.org/10.1007/s10639-021-10458-7.
Kraglund-Gauthier, W.L. and Young, D.C., 2012. Will the real John Doe stand up? Verifying the identity of online students. In L. A. Wankel and C. Wankel (Eds.), Misbehavior online in higher education (Vol. 5, pp. 355-377). England: Emerald Group Publishing Limited. https://doi.org/10.1108/S2044-9968(2012)0000005019.
Miguel, J., Caballé, S., Xhafa, F. and Prieto, J., 2014. Security in online learning assessment towards an effective trustworthiness approach to support E-learning teams. Proceedings of 2014 IEEE 28th International Conference on Advanced Information Networking and Applications: IEEE AINA 2014 (pp. 123-130), 13-16 May 2014, University of Victoria, Victoria, Canada. IEEE. http://doi.org/10.1109/aina.2014.106.
Moore, H.P., Head, J.D. and Griffin, R.B. 2017. Impeding students' efforts to cheat in online classes. Journal of Learning in Higher Education, 13(1), pp.9-23. [online] Available at: <https://files.eric.ed.gov/fulltext/EJ1139692.pdf> [Accessed 10 February 2022].
Nie, D., Panfilova, E., Samusenkov, V. and Mikhaylov, A., 2020. E-learning financing models in Russia for sustainable development. Sustainability, 12(11), pp.4412-4426. http://doi.org/10.3390/su12114412.
Nigam, A., Pasricha, R., Singh, T. and Churi, P., 2021. A systematic review on ai-based proctoring systems: Past, present and future. Education and Information Technologies, 26(5), pp.6421-6445. https://doi.org/10.1007/s10639-021-10597-x.
Prathish, S. Narayanan, S.A. and Bijlani, K., 2016. An intelligent system for online exam monitoring. Proceedings of the 2016 International Conference on Information Science (ICIS) (pp. 138-143), 12-13 August 2016, Kochi, India. IEEE. https://doi.org/10.1109/infosci.2016.7845315.
ProctorU., 2021a. A human-centered proctoring policy. [online] Available at: <https://www.proctoru.com/human-centered-proctoring> [Accessed 10 February 2022].
ProctorU., 2021b. ProctorU to discontinue exam integrity services that rely exclusively on AI. [online] Available at: <https://www.proctoru.com/industry-news-and-notes/proctoru-to-discontinue-exam-integrity-services-that-rely-exclusively-on-ai> [Accessed 10 February 2022].
Raj, R.V., Narayanan, S.A. and Bijlani, K., 2015. Heuristic-based automatic online proctoring system. Proceedings of the 2015 IEEE 15th International Conference on Advanced Learning Technologies (pp. 458-459), 6-9 July 2015, Hualien, Taiwan. IEEE. https://doi.org/10.1109/ICALT.2015.127.
Raman, R., Sairam, B., Veena, G., Vachharajani, H. and Nedungadi, P., 2021. Adoption of online proctored examinations by university students during COVID-19: Innovation diffusion study. Education and Information Technologies. [Advance online publication]. https://doi.org/10.1007/s10639-021-10581-5.
Randhavane, T., Bhattacharya, U., Kapsaskis, K., Gray, K., Bera, A. and Manocha, D. 2019. The liar's walk: Detecting deception with gait and gesture. ArXiv Preprint. [online] Available at: <https://arxiv.org/pdf/1912.06874.pdf> [Accessed 10 February 2022].
Raza, S.A., Qazi, W., Khan, K.A. and Salam, J., 2021. Social isolation and acceptance of the learning management system (LMS) in the time of COVID-19 pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 59(2), pp.183-208. https://doi.org/10.1177/0735633120960421.
Safe Exam Browser (SEB)., 2021. Developer documentation - Integration. [online] Available at: <https://safeexambrowser.org/developer/seb-integration.html> [Accessed 10 February 2022].
Slusky, L., 2020. Cybersecurity of online proctoring systems. Journal of International Technology and Information Management, 29(1), pp.56-83. [online] Available at: <https://scholarworks.lib.csusb.edu/jitim/vol29/iss1/3> [Accessed 10 February 2022].
Tayan, B.M., 2017. Academic misconduct: An investigation into male students’ perceptions, experiences & attitudes towards cheating and plagiarism in a Middle Eastern university context. Journal of Education and Learning, 6(1), pp.158-166. http://doi.org/10.5539/jel.v6n1p158.
UNESCO., 2021. One year into COVID-19 education disruption: Where do we stand? [online] Available at: <https://en.unesco.org/news/one-year-covid-19-education-disruption-where-do-we-stand> [Accessed 10 February 2022].
UNICEF., 2020. COVID-19: Are children able to continue learning during school closures?: A global analysis of the potential reach of remote learning policies. [online] Available at: <https://data.unicef.org/resources/remote-learning-reachability-factsheet/> [Accessed 10 February 2022].
Van Houdt, G., Mosquera, C. and Nápoles, G., 2020. A review on the long short-term memory model. Artificial Intelligence Review, 53(8), pp.5929-5955. https://doi.org/10.1007/s10462-020-09838-1.
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Adiy Tweissi, Wael Al Etaiwi, Dalia Al Eisawi
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Open Access Publishing
The Electronic Journal of e-Learning operates an Open Access Policy. This means that users can read, download, copy, distribute, print, search, or link to the full texts of articles, crawl them for indexing, pass them as data to software, or use them for any other lawful purpose, without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself. The only constraint on reproduction and distribution, and the only role for copyright in this domain, is that authors control the integrity of their work, which should be properly acknowledged and cited.
