Leveraging xAPI in a Corporate e-Learning Environment to Facilitate the Tracking, Modelling, and Predictive Analysis of Learner Behaviour

Authors: Libor Zachoval, Daire O Broin, Oisin Cawley

Abstract:

E-learning platforms, such as Blackboard have two major shortcomings: limited data capture as a result of the limitations of SCORM (Shareable Content Object Reference Model), and lack of incorporation of Artificial Intelligence (AI) and machine learning algorithms which could lead to better course adaptations. With the recent development of Experience Application Programming Interface (xAPI), a large amount of additional types of data can be captured and that opens a window of possibilities from which online education can benefit. In a corporate setting, where companies invest billions on the learning and development of their employees, some learner behaviours can be troublesome for they can hinder the knowledge development of a learner. Behaviours that hinder the knowledge development also raise ambiguity about learner’s knowledge mastery, specifically those related to gaming the system. Furthermore, a company receives little benefit from their investment if employees are passing courses without possessing the required knowledge and potential compliance risks may arise. Using xAPI and rules derived from a state-of-the-art review, we identified three learner behaviours, primarily related to guessing, in a corporate compliance course. The identified behaviours are: trying each option for a question, specifically for multiple-choice questions; selecting a single option for all the questions on the test; and continuously repeating tests upon failing as opposed to going over the learning material. These behaviours were detected on learners who repeated the test at least 4 times before passing the course. These findings suggest that gauging the mastery of a learner from multiple-choice questions test scores alone is a naive approach. Thus, next steps will consider the incorporation of additional data points, knowledge estimation models to model knowledge mastery of a learner more accurately, and analysis of the data for correlations between knowledge development and identified learner behaviours. Additional work could explore how learner behaviours could be utilised to make changes to a course. For example, course content may require modifications (certain sections of learning material may be shown to not be helpful to many learners to master the learning outcomes aimed at) or course design (such as the type and duration of feedback).

Keywords: Compliance Course, Corporate Training, Learner Behaviours, xAPI.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.3593130

References:

[1] Gibson, M., 2013. Does SCORM prevent good education? (WWW Document). URL http://markgibson.info/2013/07/does-scorm-prevent-good-education.html (accessed 3.20.18).
[2] Graf, S., List, B., 2005. An evaluation of open source e-learning platforms stressing adaptation issues. IEEE, pp. 163–165. https://doi.org/10.1109/ICALT.2005.54
[3] Aleven, V., Sewall, J., Popescu, O., Xhakaj, F., Chand, D., Baker, R., Wang, Y., Siemens, G., Rosé, C., Gasevic, D., 2015. The Beginning of a Beautiful Friendship? Intelligent Tutoring Systems and MOOCs, in: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (Eds.), Artificial Intelligence in Education. Springer International Publishing, Cham, pp. 525–528. https://doi.org/10.1007/978-3-319-19773-9_53
[4] My e-Learning World, 2016. SCORM vs Tin Can vs AICC: Which Best Suits Your Needs? (WWW Document). My Blog. URL https://myelearningworld.com/scorm-vs-tin-can-vs-aicc-the-lms-standard-showdown/ (accessed 3.20.18).
[5] Grossman, R., Salas, E., 2011. The transfer of training: what really matters: The transfer of training. International Journal of Training and Development 15, 103–120. https://doi.org/10.1111/j.1468-2419.2011.00373.x
[6] Little, B., 2014. Best practices to ensure the maximum ROI in learning and development. Industrial and Commercial Training 46, 400–405. https://doi.org/10.1108/ICT-08-2014-0051
[7] Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z., 2004. Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game the System,” in: In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. p. 8.
[8] Baker, R.S.J.D., Corbett, A.T., Roll, I., Koedinger, K.R., 2008. Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction 18, 287–314. https://doi.org/10.1007/s11257-007-9045-6
[9] Bailey, W., 2005. standards briefings series 4.
[10] Parr, D., 2016. SCORM – Set of Standards that make Courses and LMS Compatible. Paradiso eLearning Blog.
[11] Rustici Software, 2018b. One Minute SCORM Overview for Anyone - SCORM – (WWW Document). SCORM -. URL https://scorm.com/ scorm-explained/one-minute-scorm-overview/ (accessed 3.5.18).
[12] Rustici Software, 2018c. SCORM Explained (WWW Document). SCORM -. URL https://scorm.com/scorm-explained/ (accessed 3.20.18).
[13] Rustici Software, 2018a. Benefits of SCORM - SCORM – (WWW Document). SCORM -. URL https://scorm.com/scorm-explained/ business-of-scorm/benefits-of-scorm/ (accessed 3.20.18).
[14] Glahn, C., 2013, September. Using the adl experience api for mobile learning, sensing, informing, encouraging, orchestrating. In Next Generation Mobile Apps, Services and Technologies (NGMAST), 2013 Seventh International Conference on (pp. 268-273). IEEE.
[15] Bakharia, A., Kitto, K., Pardo, A., Gašević, D. and Dawson, S., 2016, April. Recipe for success: lessons learnt from using xAPI within the connected learning analytics toolkit. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 378-382). ACM.
[16] Long, R., Murphy, J., Newton, C., Hruska, M., Medford, A., Kilcullen, T. and Harvey Jr, R., 2015. Adapting gunnery training using the experience API. In Proceedings of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC).
[17] Dodero, J.M., González-Conejero, E.J., Gutiérrez-Herrera, G., Peinado, S., Tocino, J.T. and Ruiz-Rube, I., 2017. Trade-off between interoperability and data collection performance when designing an architecture for learning analytics. Future Generation Computer Systems, 68, pp.31-37.
[18] Conati, C., 2009. Intelligent Tutoring Systems: New Challenges and Directions., in: IJCAI. Presented at the Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), Morgan Kaufmann, San Francisco, pp. 2–7.
[19] Ghali, M.A., Ayyad, A.A., Abu-Naser, S.S., Laban, M.A., 2018. An Intelligent Tutoring System for Teaching English Grammar. International Journal of Academic Engineering Research (IJAER) 2, 1–6.
[20] Paquette, L., de Carvalho, A.M., Baker, R.S., 2014. Towards Understanding Expert Coding of Student Disengagement in Online Learning. CogSci 6.
[21] Paquette, L., Baker, R.S., 2019. Comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. Interactive Learning Environments 1–13. https://doi.org/10.1080/10494820.2019.1610450
[22] Aleven, V., Mclaren, B.M., Roll, O., Koedinger, K., Aleven, V., Mclaren, B., Roll, I., Koedinger, K., 2004. Toward Tutoring Help Seeking: Applying Cognitive Modeling to Meta-Cognitive Skills, in: In Proceedings of the 7th International Conference on Intelligent Tutoring Systems. Springer-Verlag.