Evaluating lab assistant chatbot on student learning and behaviors in a programming short course
2
Issued Date
2026-06-01
Resource Type
eISSN
2666920X
Scopus ID
2-s2.0-105025094025
Journal Title
Computers and Education Artificial Intelligence
Volume
10
Rights Holder(s)
SCOPUS
Bibliographic Citation
Computers and Education Artificial Intelligence Vol.10 (2026)
Suggested Citation
Noraset T., Supratak A., Ragkhitwetsagul C., Worathong N., Tuarob S. Evaluating lab assistant chatbot on student learning and behaviors in a programming short course. Computers and Education Artificial Intelligence Vol.10 (2026). doi:10.1016/j.caeai.2025.100527 Retrieved from: https://repository.li.mahidol.ac.th/handle/123456789/114421
Title
Evaluating lab assistant chatbot on student learning and behaviors in a programming short course
Author's Affiliation
Corresponding Author(s)
Other Contributor(s)
Abstract
The rise of generative AI has increased interest in its application as an intelligent lab assistant in programming education, but concerns persist over its educational value and potential exploitation. While previous work supports using a customized chatbot as an assistant that provides specific guidance rather than allowing students to prompt responses freely, empirical evidence directly comparing these approaches is still lacking. This study evaluates the impact of two chatbot designs, Unrestricted and Assistant, on student learning and behavior in a short Python programming course. Through a controlled experiment involving 42 participants, we found that students using the Assistant chatbot, which provided guidance through preset and free-text prompts without offering direct solutions, showed significantly greater improvement from pre- to post-test than those using an Unrestricted chatbot. Analysis of over 1000 chatbot interactions revealed a strong preference for free-text input and a high rate of attempted exploits among participants. Additionally, prompt injection tests demonstrated the Assistant chatbot’s partial vulnerability to hijacking attempts. These findings highlight the benefits and limitations of AI assistants in programming education, underscoring the importance of guided interaction design to support learning while minimizing exploitation.