A Programming Assessment Software Artefact Enhanced with the Help of Learners

Authors: Romeo A. Botes, Imelda Smit

Abstract:

The demands of an ever changing and complex higher education environment, along with the profile of modern learners challenge current approaches to assessment and feedback. More learners enter the education system every year. The younger generation expects immediate feedback. At the same time, feedback should be meaningful. The assessment of practical activities in programming poses a particular problem, since both lecturers and learners in the information and computer science discipline acknowledge that paper-based assessment for programming subjects lacks meaningful real-life testing. At the same time, feedback lacks promptness, consistency, comprehensiveness and individualisation. Most of these aspects may be addressed by modern, technology-assisted assessment. The focus of this paper is the continuous development of an artefact that is used to assist the lecturer in the assessment and feedback of practical programming activities in a senior database programming class. The artefact was developed using three Design Science Research cycles. The first implementation allowed one programming activity submission per assessment intervention. This pilot provided valuable insight into the obstacles regarding the implementation of this type of assessment tool. A second implementation improved the initial version to allow multiple programming activity submissions per assessment. The focus of this version is on providing scaffold feedback to the learner – allowing improvement with each subsequent submission. It also has a built-in capability to provide the lecturer with information regarding the key problem areas of each assessment intervention.

Keywords: Programming, computer-aided assessment, technology-assisted assessment, programming assessment software, design science research, mixed-method.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1339622

References:

[1] Staley, D.J. and D.A. Trinkle, The changing landscape of higher education. Educause Review, 2011. 46(1): p. 16-33.
[2] Draft, S., Computer Science Curricula 2013. 2012.
[3] Attaway, S., Matlab: A Practical Introduction to Programming and Problem Solving. 2009: Elsevier Science.
[4] Cormen, T.H., et al., Introduction to Algorithms. 2001: MIT Press.
[5] Deitel, H.M. and P.J. Deitel, Visual Basic 2005: How to Program. 2006: Pearson/Prentice Hall.
[6] Deitel, P.J. and H.M. Deitel, Visual C# 2012: How to Program. 2013: Pearson Education, Limited.
[7] Brown, S.A. and A. Glasner, Assessment matters in higher education: Choosing and using diverse approaches. 1999: McGraw-Hill International.
[8] Hui, W., et al., Technology‐assisted learning: a longitudinal field study of knowledge category, learning effectiveness and satisfaction in language learning. Journal of Computer Assisted Learning, 2008. 24(3): p. 245-259.
[9] Jenkins, M., Unfulfilled Promise: formative assessment using computer-aided assessment. Learning and Teaching in Higher Education, 2004. 1(1): p. 67-80.
[10] Charman, D., Issues and impacts of using computer-based assessments (CBAs) for formative assessment. S. Brown, J. Bull and P. Race (Eds.), Computer-Assisted Assessment in Higher Education, 1999: p. 85-94.
[11] Sadler, D.R., Formative assessment and the design of instructional systems. Instructional science, 1989. 18(2): p. 119-144.
[12] Garrison, D.R., E-learning in the 21st century: A framework for research and practice. 2011: Taylor & Francis.
[13] Gregor, S. and A.R. Hevner, Positioning and presenting design science research for maximum impact. MIS Quaterly, 2013. 37(2): p. 337-355.
[14] Hevner, A.R. and S. Chatterjee, Design Research in Information Systems. Integrated Series in Information Systems, ed. R. Sharda and S. Vob. Vol. 22. 2010, New York Dordrecht Heidelberg London: Springer. 320.
[15] Hevner, A.R., et al., Design Science in Information Systems Research. MIS Quaterly, 2004. 28(1): p. 75-105.
[16] Peffers, K., et al., A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 2008. 24(3): p. 45-77.
[17] Vaishnavi, V. and K. Kuechler. Design Research in Information Systems. 2004 16 August 2009 (cited 2013 14 June); Available from: http://desrist.org/design-research-in-information-systems/.
[18] Iivari, J., A paradigmatic analysis of information systems as a design science. Scandinavian Journal of Information Systems, 2007. 19(2): p. 39.
[19] Saunders, M., P. Lewis, and A. Thornhill, Research methods for business students. 5th ed. 2009, Harlow England: Pearson. 614.
[20] Oates, B.J., Researching Information Systems and Computing. 2006: SAGE Publications.
[21] Seaman, C.B., Qualitative methods in empirical studies of software engineering. Software Engineering, IEEE Transactions on, 1999. 25(4): p. 557-572.