Keywords
diagnostic
generative artificial intelligence
How to Cite
License
Copyright (c) 2025 Jan-Martin Geiger, Anna Wenzel
This work is licensed under a Creative Commons Attribution 4.0 International License.
Abstract
Within the scope of this study, a questionnaire instrument is developed to assess subjective learning experiences in the context of learning with natural language AI-based conversational agents. To this end, a questionnaire on subjective learning was constructed by synthesizing existing scales and subsequently validated in two learning scenarios. These scenarios involved the implementation of two economic serious games, during which an AI-based conversational agent, specifically tailored to the content of the sessions, was utilized to provide individualized support to students throughout their learning process. An initial exploratory factor analysis revealed five factors relevant to learning with conversational agents, represented by 17 items. This study contributes to the diagnostic repertoire for examining learners’ perceptions of AI-supported pedagogical tools. From a scientific perspective, it enhances methodological approaches, while on a practical pedagogical level, it offers novel insights for advancing learning support and evaluation strategies.
References
Ait Baha, Tarek, Mohamed El Hajji, Youssef Es-Saady, und Hammou Fadili. 2024. «The impact of educational chatbot on student learning experience». Education and Information Technologies, 29(8), 10153-10176. https://www.doi.org/10.1007/s10639-023-12166-w.
Arndt, Holger. 2020. Ökonomische Bildung. Erlangen: FAU University Press.
Barz, Nathalie, Manuela Benick, Laura Dörrenbächer-Ulrich, und Franziska Perels. 2024. «The effect of digital game-based learning interventions on cognitive, metacognitive, and affective-motivational learning outcomes in school: A meta-analysis». Review of Educational Research, 94(2), 193–227. https://doi.org/10.3102/00346543231167795.
Bavaresco, Rodrigo, Diórgenes Silveira, Eduardo Reis, Jorge Barbosa, Rodrigo Righi, Cristiano Costa, … Mariangela Vanzin. 2020. «Conversational agents in business: A systematic literature review and future research directions». Computer Science Review, 36, 100239. https://doi.org/10.1016/j.cosrev.2020.100239.
Baylor, Amy L., und Yanghee Kim. 2005. «Simulating instructional roles through pedagogical agents». International Journal of Artificial Intelligence in Education, 15(2), 95–115. https://doi.org/10.1007/s40593-015-0055-y.
Bellotti, Francesco, Riccardo Berta, Alessandro de Gloria, Elisa Lavagnino, Alessandra Antonaci, Francesca Dagnino, … Igor S. Mayer. 2014. «Serious games and the development of an entrepreneurial mindset in higher education engineering students». Entertainment Computing, 5(4), 357–66. https://doi.org/10.1016/j.entcom.2014.07.003.
Borsci, Simone, Alessio Malizia, Martin Schmettow, Frank van der Velde, Gunay Tariverdiyeva, Divyaa Balaji, und Alan Chamberlain. 2022. «The chatbot usability scale: the design and pilot of a usability scale for interaction with AI-based conversational agents». Personal and Ubiquitous Computing, 26, 95–119. https://doi.org/10.1007/s00779-021-01582-9.
Borsci, Simone, Martin Schmettow, Alessio Malizia, Alan Chamberlain, und Frank van der Velde. 2023. «A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation». Personal and Ubiquitous Computing, 27(2), 317–30. https://www.doi.org/10.1007/s00779-022-01690-0.
Castro-Alonso, Juan C., Rachel M. Wong, Olusola O. Adesope, und Fred Paas. 2021. «Effectiveness of multimedia pedagogical agents predicted by diverse theories: A meta-analysis». Educational Psychology Review, 33, 989–1015. https://doi.org/10.1007/s10648-020-09587-1.
Chavez Rojas, Jorge, Faure Ninoles, Jaime und Barril Madrid, und Juan Pablo. 2023. «The construction of teachers’ professional identity: An analysis of subjective learning experiences». European Journal of Teacher Education, 46(2), 256–73. https://doi.org/10.1080/02619768.2021.1905627.
Chen, Li, Dirk Ifenthaler, und Jane Yau. 2021. «Online and blended entrepreneurship education: a systematic review of applied educational technologies». Entrepreneurship Education, 4(2), 191–232. https://www.doi.org/10.1007/s41959-021-00047-7.
Clarebout, Geraldine, Jan Elen, W Lewis Johnson, und Erin Shaw. 2002. «Animated pedagogical agents: An opportunity to be grasped?» Journal of Educational multimedia and hypermedia, 11(3), 267–86.
Clark, Douglas B., Emily E. Tanner-Smith, und Stephen S. Killingsworth. 2016. «Digital games, design, and learning: A systematic review and meta-analysis». Review of Educational Research, 86(1), 79–122. https://doi.org/10.3102/00346543155820.
Davis, Fred D., Richard P. Bagozzi, und Paul R. Warshaw. 1989. «User Acceptance of Computer Technology: A Comparison of Two Theoretical Models». J Manag Sci, 35(8), 982–1003.
Döring, Nicola, und Jürgen Bortz. 2016. Forschungsmethoden und Evaluation. Wiesbaden: Springer. https://doi.org/10.1007/978-3-642-41089-5.
Faisal, Nadia, Mehmood Chadhar, Anitra Goriss-Hunter, und Andrew Stranieri. 2022. «Business simulation games in higher education: A systematic review of empirical research». Human Behavior and Emerging Technologies, 2022(1), 1578791. https://doi.org/10.1155/2022/1578791.
Fox, Joe, Luke Pittaway, und Ikenna Uzuegbunam. 2018. «Simulations in Entrepreneurship Education: Serious Games and Learning Through Play». Entrepreneurship Education and Pedagogy, 1(1), 61–89. https://www.doi.org/10.1177/2515127417737285.
Gimpel, Henner, Kristina Hall, Stefan Decker, Torsten Eymann, Luis Lämmermann, Alexander Mädche, … Steffen Vandirk. 2023. Unlocking the Power of Generative AI Models and Systems such as GPT-4 and ChatGPT for Higher Education. A Guide for Students and Lecturers.
Haken, Hermann, und Günter Schiepek. 2006. Synergetik in der Psychologie. Selbstorganisation verstehen und gestalten. Göttingen: Hogrefe.
Han, Jining, Geping Liu, und Yuxin Gao. 2023. «Learners in the Metaverse: A systematic review on the use of roblox in learning». Education Sciences, 13(3), 296. https://doi.org/10.3390/educsci13030296.
Han, Songhee, Xiaofen Hamilton, Ying Cai, Peixia Shao, und Min Liu. 2023. «Knowledge-based chatbots: a scale measuring students’ learning experiences in massive open online courses». Educational technology research and development, 71(6), 2431–56. https://www.doi.org/10.1007/s11423-023-10280-7.
Hew, Khe Foon, Weijiao Huang, Jiahui Du, und Chengyuan Jia. 2021. Using Chatbots in Flipped Learning Online Sessions: Perceived Usefulness and Ease of Use. Cham: Springer Nature.
Hobert, Sebastian, und Raphael Meyer von Wolff. 2019. Say hello to your new automated tutor – a structured literature review on pedagogical conversational agents. Conference Paper.
Hochmuth, Jörg und Kirchner, Vera. 2019/20. Einstellungen von Wirtschaft unterrichtenden Lehrpersonen zu Fortbildungen in diesem Feld: Ergebnisse eines Online-Surveys im Rahmen des ELF-Projekts.
Klar, Maria, und Johannes Schleiss. 2024. «Künstliche Intelligenz im Kontext von Kompetenzen, Prüfungen und Lehr-Lern-Methoden: Alte und neue Gestaltungsfragen». MedienPädagogik: Zeitschrift für Theorie und Praxis der Medienbildung 58: 41–57. https://doi.org/10.21240/mpaed/58/2024.03.24.X.
König, Alexander. 2024. «KI-Tools für Schule und Unterricht». In Praxisratgeber: Künstliche Intelligenz als Unterrichtsassistent. Wie KI-Tools das Lehrerleben erleichtern, herausgegeben von Alexander König und Julia Mosbach. Hannover: Friedrich.
Kulkarni, Pradnya, Ameya Mahabaleshwarkar, Mrunalini Kulkarni, Nachiket Sirsikar, und Kunal Gadgil. 2019. «Conversational AI: An overview of methodologies, applications & future scope». Paper presented at the 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA). https://doi. org/10.1109/ICCUBEA47591.2019.9129347.
Landes, Tom und Matthias Ziegler. 2015. Ein Praxisbeispiel zur Konstruktion eines Lehrevaluationsinstruments: Berliner Lehrevaluationsinventar für Vorlesungen (BLEI-VL) (Vol. 61): Göttingen: Hogrefe.
Liening, Andreas. 2019. Ökonomische Bildung: Grundlagen und neue synergetische Ansätze (2. Aufl.). Wiesbaden: Springer Gabler. https://doi.org/10.1007/978-3-658-24731-7.
Liening, Andreas, und Ewald Mittelstädt. 2011. «Börsen-Planspiele – Leuchttürme oder Irrlichter der Finanziellen Allgemeinbildung?» In Finanzielle Bildung in der Schule. Mündige Verbraucher durch Konsumentenbildung, herausgegeben von Thomas Retzmann, 99–114. Schwalbach i. Ts.: Wochenschau.
Liening, Andreas, Guido Strunk, und Ewald Mittelstädt. 2013. «Management Learning in Times of Crisis: An Experimental Study based on Synergetics». Nonlinear dynamics, psychology, and life sciences, 17(4), 517–41.
Maier, Uwe. 2010. «Formative Assessment – Ein erfolgversprechendes Konzept zur Reform von Unterricht und Leistungsmessung?» Zeitschrift für Erziehungswissenschaft 13 (2), 293–308. https://doi.org/10.1007/s11618-010-0124-9.
Miner, Adam S., Arnold Milstein, Stephen Schueller, Roshini Hegde, Christina Mangurian, und Eleni Linos. 2016. «Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health». JAMA internal medicine, 176(5), 619–25. https://doi.org/10.1001/jamainternmed.2016.0400.
Montenegro, Joao Luis Zeni, Cristiano André da Costa, und Rodrigo da Rosa Righi. 2019. «Survey of conversational agents in health». Expert Systems with Applications 129, 56–67. https://doi.org/10.1016/j.eswa.2019.03.054.
Pekrun, Reinhard, Thomas Goetz, Anne C. Frenzel, Petra Barchfeld, und Raymond P. Perry. 2011. «Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ)». Contemporary educational psychology 36 (1), 36–48. https://doi.org/10.1016/j.cedpsych.2010.10.002.
Pekrun, Reinhard, Elisabeth Vogl, Krista R. Muis, und Gale M. Sinatra. 2017. «Measuring emotions during epistemic activities: the Epistemically-Related Emotion Scales». Cognition and Emotion, 31(6), 1268–76. https://doi.org/10.1080/02699931.2016.1204989.
Pintrich, Paul R., David A. F. Smith, Teresa Garcia, und Wilbert J. McKeachie. 1991. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor: National Center for Research to Improve Postsecondary Teaching and Learning.
Plass, Jan L, Bruce D. Homer, und Charles K. Kinzer. 2015. «Foundations of game-based learning». Educational psychologist 50 (4): 258–83. http://doi.org/10.1080/00461520.2015.1122533.
Plass, Jan L., Bruce D. Homer, Richard E. Mayer, und Charles K. Kinzer. 2020. Theoretical Foundations of Game-Based and Playful Learning: Mit Press.
Platz, Liane. 2022. «Learning with serious games in economics education a systematic review of the effectiveness of game-based learning in upper secondary and higher education». International journal of educational research, 115, 102031. https://doi.org/10.1016/j.ijer.2022.102031.
Quah, Jon TS, und YW Chua. 2019. «Chatbot assisted marketing in financial service industry». Paper presented at the Services Computing–SCC 2019: 16th International Conference, Held as Part of the Services Conference Federation, SCF 2019, San Diego, CA, USA, June 25–30, 2019, Proceedings 16. https://doi.org/10.1007/978-3-030-23554-3_8.
Reeves, Byron, und Clifford Nass. 1996. The media equation: How people treat computers, television, and new media like real people and places. Center for the Study of Language and Information, Cambridge University Press.
Reiter, Ehud, und Robert Dale,. 1997. «Building applied natural language generation systems». Natural Language Engineering, 3(1), 57–87. https://doi.org/10.1017/S1351324997001502.
Saad, Syed, Gary W. Carter, Mark Rothenberg, und Enid Israelson. 1999. Testing and Assessment: An Employer‘s Guide to Good Practices.
Schiepek, Günter, Wolfgang Aichhorn, und Guido Strunk. 2012. «Der Therapie-Prozessbogen (TPB) – Faktorenstruktur und psychometrische Daten». Zeitschrift für Psychosomatische Medizin und Psychotherapie 58 (3): 257–66. https://doi.org/10.13109/zptm.2012.58.3.257.
Schlimbach, Ricarda, Bijan Khosrawi-Rad, Tim C. Lange, Timo Strohmann, und Susanne Robra-Bissantz. 2024. «Design Knowledge for Virtual Learning Companions from a Value-centered Perspective». Communications of the association for Information Systems 54 (1): 8. https://doi.org/10.17705/1CAIS.05411.
Schöbel, Sofia, Anuschka Schmitt, Dennis Benner, Mohammed Saqr, Andreas Janson, und Jan Marco Leimeister. 2023. «Charting the Evolution and Future of Conversational Agents: A Research Agenda Along Five Waves and New Frontiers». Information Systems Frontiers, 1–26.
Schroeder, Noah L., Robert O. Davis, und Eunbyul Yang. 2025. «Designing and learning with pedagogical agents: An umbrella review». Journal of Educational Computing Research 62 (8): 1907-36. https://doi.org/10.1177/07356331241288476.
Sender, Till. 2017. Wirtschaftsdidaktische Lerndiagnostik und Komplexität. Lokalisierung liminaler Unsicherheitsphasen im Hinblick auf Schwellenübergänge. Wiesbaden: Springer Gabler. https://doi.org/10.1007/978-3-658-18947-1.
Sikström, Pieta, Chiara Valentini, Anu Sivunen, und Tommi Kärkkäinen. 2022. «How pedagogical agents communicate with students: A two-phase systematic review». Computers & Education, 188, 104564. https://doi.org/10.1016/j.compedu.2022.104564.
Streiner, David L., Geoffrey R. Norman, und John Cairney. 2015. Health measurement scales: a practical guide to their development and use. Oxford: Oxford University Press. https://doi.org/10.1093/med/9780199685219.001.0001.
Vogl, Elisabeth, Reinhard Pekrun, Kou Murayama, Kristina Loderer, und Sandra Schubert. 2019. «Surprise, Curiosity, and Confusion Promote Knowledge Exploration: Evidence for Robust Effects of Epistemic Emotions». Frontiers in Psychology, 10 (2474). https://doi.org/10.3389/fpsyg.2019.02474.
Weber, Florian, Thiemo Wambsganss, Dominic Rüttimann, und Matthias Söllner. 2021. «Pedagogical agents for interactive learning: A taxonomy of conversational agents in education». Paper presented at the Forty-Second International Conference on Information Systems. Austin, Texas.
Weyland, Michael. 2016. Experimentelles Lernen und Ökonomische Bildung. Ein Beitrag zur fachdidaktischen Entwicklungsforschung. Wiesbaden: Springer VS. https://doi.org/10.1007/978-3-658-14611-5.
Wiegand, Dorothee. 2024. «Klett stellt neuen KI-Chatbot für Lehrer vor». Retrieved 04.09.2024, from https://www.heise.de/news/Klett-stellt-neuen-KI-Chatbot-fuer-Lehrer-vor-9854872.html.
Wouters, Pieter, Christof van Nimwegen, Herre van Oostendorp, und Erik D. van Der Spek. 2013. «A meta-analysis of the cognitive and motivational effects of serious games». Journal of Educational Psychology, 105 (2), 249–65. https://doi.org/10.1037/a0031311.
Zwick, William R., und Wayne F. Velicer. 1986. «Comparison of Five Rules for Determining the Number of Components to Retain». Psychological Bulletin, 99 (3), 432–42. https://doi.org/10.1037/0033-2909.99.3.432.