Academic Digital Library's Evaluation Criteria: User-Centered Approach

Authors: Razilan A. Kadir, Wan A. K. W. Dollah, Fatimah A. Saaid, S. Diljit

Abstract:

Academic digital libraries emerged as a result of advances in computing and information systems technologies, and had been introduced in universities and to public. As results, moving in parallel with current technology in learning and researching environment indeed offers myriad of advantages especially to students and academicians, as well as researchers. This is due to dramatic changes in learning environment through the use of digital library system which giving spectacular impact on these societies- way of performing their study/research. This paper presents a survey of current criteria for evaluating academic digital libraries- performance. The goal is to discuss criteria being applied so far for academic digital libraries evaluation in the context of user-centered design. Although this paper does not comprehensively take into account all previous researches in evaluating academic digital libraries but at least it can be a guide in understanding the evaluation criteria being widely applied.

Keywords: Academic digital libraries, evaluation criteria, performance, user-centered.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1330261

References:

[1] A. Mohd Amin, L. Abdul Wahab & R.Abdul Aziz. Roles and functions of information professionals in digital libraries. Kekal Abadi, 22 (2), 2003, pp.7-13.
[2] J.C. Bertot, C.R. McClure & E.M. William. Assessing Digital Libraries: Evaluation Strategies, Practices, and Issues. Proceedings of the Annual Meeting- American Society for Information Science, vol .41, p:561, 2004. (Online) http://www.kc.tsukuba.ac.jp/dlkc/eproceedings/ papers/dlkc04pp72.pdf
[3] J.C. Bertot, J.T. Snead, P.T. Jaeger and C.R. McClure. Functionality, usability, and Accessibility Iterative user-centered evaluation strategies for digital libraries. Performance Measurement and Metrics, vol.7, no.1, 2006, pp. 17-28.
[4] A. P. Bishop, L. J. Neumann, S. L. Star, C. Merkel, E. Ignacio & R. J. Sandusky. Digital libraries: Situating use in changing information infrastructure. Journal of the American Society for Information Science, 51, 2000, pp.394-413.
[5] A. Blandford, & G. Buchanan. Workshop report: Usability of digital libraries, JCDL-02. SIGIR Forum, vol.36, no.2, pp.83-89, 2002.
[6] C. L. Borgman, G. H. Leazer, A.J. Gilliland-Swetland & R. Gazan. Iterative design and evaluation of a geographic digital library for university students: A case study of the Alexandria Digital Earth Prototype (ADEPT). Proceedings of the 5th European Conference on Research and Advanced Technology for Digital Libraries. 2001, pp. 390−401. London: Springer-Verlag.
[7] C. L. Borgman, G. H. Leazer, A. J. Gilliland-Swetland & R. Gazan. Evaluating digital libraries for teaching and learning in undergraduate education: A case study of the Alexandria digital earth prototype (ADEPT). Library Trends, 49, 2000, pp.228-250.
[8] N. Bryan-Kinns, & A. Blandford, A. Understanding Interaction in Digital Libraries: a case study on applying Interaction Framework. RIDL working paper, 2000.
[9] B. Buttenfield. Usability evaluation of digital libraries. Science and Technology Libraries, 17(3/4), 1999, pp. 39-50.
[10] D.S. Carter & J. Janes. Unobtrusive data analysis of digital reference questions and service at the Internet Public Library: An exploratory study. Library Trends, vol.49, 2000, pp.251-265.
[11] L. Champeny, C.L. Borgman, G.H. Leazer, A.J. Gilliland-Swetland, K.A. Millwood, L. D-Avolio, J.R. Finley, L.J. Smart, P.D. Mautone, R.E. Mayer and R.A. Johnson. Developing a Digital Learning Environment: An Evaluation of Design and Implementation Processes. JCDL-04, June 7-11, 2004, Tucson, Arizona, USA, 2004, pp.37-46.
[12] G.G. Chowdhury, & S. Chowdhury. Introduction to digital libraries. London: Facet Publisher, 2003.
[13] G.G. Chowdhury & S. Chowdhury. An Overview of the Information Retrieval Features of Twenty Digital Libraries. Aslib, vol.34, no.4, 2000, pp:342.
[14] S. Chowdhury, M. Landoni and F. Gibb. Usability and impact of digital libraries: a review. 2006. (online) www.emeraldinsight.com/1468- 4527.htm
[15] A. Coleman and T. Sumner . Digital Libraries and User Needs: Negotiating the Future. Journal of Digital Information, vol.5, no.3. 2004.(Online) http://jodi.tamu.edu/Articles/v05/i03/editorial/#Wattenberg98.
[16] B. Dalgarno. Interpretations of constructivism and consequences for computer assisted learning. British J. Educational Technology, vol.32, 2001, pp. 183-194.
[17] B. Dervin & M. Nilan. "Information needs and uses". In M. Williams (Ed.) Annual Review of Information Science and Technology, vol.21, 1986, pp.3-33, White Plains, NY: Knowledge Industries.
[18] A. Dillon. Evaluating on TIME: A framework for the expert evaluation of digital library interface usability. International Journal on Digital Libraries, 2(2/3), 1999, pp. 170-177.
[19] N. Fuhr, P. Hansen, M. Mabe, A. Micsik, & I. So¨ lvberg. Digital libraries: a generic classification and evaluation scheme. In Constantopoulos, P.& So¨lvberg, I. (Eds.), Proceedings of ECDL-01, 2001, pp. 187-199. Darmstadt, Germany.
[20] D.L. Gard. Digital Libraries and the Environment: A Comparative Life Cycle Energy Analysis. Master thesis. University of Michigan, MI. 2001.
[21] D.H-L. Goh, A. Chua, D.A. Khoo, E.B-H. Khoo, E.B-T. Mak, M.W-N. Ng. A checklist for evaluating open source digital library software. Online Information Review, vol.30, no.4, 2006, pp.360-379.
[22] M.A. Goncalves, B.L. Moreira, E.A. Fox, and L.T. Watson. ÔÇÿÔÇÿWhat is a good digital library?--- A quality model for digital libraries. Information Processing and Management, vol. 43, 2007, pp.1416-1437.
[23] S.J. Hammill. (2003). Usability testing at Florida International University Libraries: what we learned", Electronic Journal of Academic and Special Librarianship, vol.4, no.1, (Online): http://southernlibrarianship.icaap.org/content/v04n01/Hammill_s01.htm
[24] L. L. Hill, L. Carver, M. Larsgaard, R. Dolin, T. R. Smith, J. Frew. Alexandria digital library: User evaluation studies and system design. Journal of the American Society for Information Science, vol. 51, 2000, pp. 246-259.
[25] N. Huling. "References services and information access". In Schement, J.R. (ed.). Encyclopedia of communication and information. New York: Gale Group, 2002, pp. 867-874.
[26] D. Hung. Design principles for web-based learning: Implications from Vygotskian thought. Educational Technology, vol. 41, 2001, pp. 33-41.
[27] D. Hung, and M. Nichani. Constructivism and e-learning: balancing between the individual and social levels of cognition. Educational Technology, vol. 41, 2001, pp. 40-44.
[28] J. Jeng. Usability Assessment of Academic Digital Libraries: Effectiveness, Efficiency, Satisfaction, and Learnability. Libri, vol. 55, 2005a, pp. 96-121.
[29] J. Jeng. What is usability in the context of the digital library and how can it be measured? Information Technology and Libraries, vol. 24, no.2, 2005b, pp. 47-56.
[30] L. Kalinichenko. Digital Libraries in Education: Analytical Survey. Moscow, UNESCO Institute for Information Technologies in Education, 2003.(Online) http://unesdoc.unesco.org/images/0013/001310/131025EB.pdf
[31] A. A. Karoulis, and A. Andreas Pombortsis. Heuristic evaluation of Web-based ODL programs. In "Usability evaluation of online learning programs", (ed.) Claude Ghaoui, 2003, pp.88-109. Hershey, PA: Information Science Publishing.
[32] A. R. C. Kassim, & T. R. Kochtanek. Designing, implementing, and evaluating an educational digital library resource. Online Information Review, vol. 27, no.3) 2003, pp. 160−168.
[33] H. H. Kim, & Y. H. Kim. Usability study of digital institutional repositories. The Electronic Library, vol. 26, no.6, 2008, pp. 863-881.
[34] C. Kuhltaul. Learning in digital libraries: An information search process approach. Library Trends, vol. 45, 1997, pp. 708-724.
[35] H. Long. An Assessment of the Current State of Digital Library Evaluation. A Master-s paper for the M.S. in I.S. degree. April, 2002.
[36] A. Lukasiewicz. Exploring the role of digital academic libraries. Library Review, vol. 56, vol.9, 2007, pp. 821-827.
[37] G. Marchionini. Evaluating Digital Libraries: A Longitudinal and Multifaceted View. Library Trends, vol. 49, no.2, 2000, pp. 304-333.
[38] G. Marchionini, C. Plaisant, & A. Komlodi. The People in Digital Libraries: Multifaceted Approaches to Assessing Needs and Impact. Library Trends, vol. 49(2), 1999, pp. 304-333.
[39] G. Marchionini, C. Plaisant, & A. Komlodi. The People in Digital Libraries: Multifaceted Approaches to Assessing Needs and Impact In Bishop, A., van House, N., and Buttenfield, B., Digital Library Use: Social Practice in Design and Evaluation pp. 119-160. Cambridge, MA: The MIT Press, 2003.
[40] B. B. Marshall, H. Chen, R. Shen, and E.A. Fox. Moving Digital Libraries into the Student Learning Space: The GetSmart Experience. ACM Journal on Educational resources in Computing, vol. 6, no.1, Article 2, 2006, pp. 2-20
[41] S. Mishra. A design framework for online learning environments. British J. Educational Technology, vol. 33, 2002, pp. 493-496.
[42] R. Mittal, and G. Mahesh. Digital libraries and repositories in India: an evaluative study. Information systems, vol. 42, no.3, 2008, pp. 286-302.
[43] A. K. Mohd Razilan, S. Fatimah Almah, S. Diljit. A Review on Digital Libraries Evolution: Information Systems and Information Retrieval Perspectives, Proceedings of CAMP-08, 18 March 2008, Kuala Lumpur, 2008, pp. 204-209.
[44] R. Oliver. Exploring strategies for online teaching and learning. Distance Education, vol. 20, 1999, pp. 240-254.
[45] T.C. Reeves, X. Apedoe, & Y. H. Woo. (2005). Evaluating Digital Libraries: A User- Friendly Guide. The University of Georgia, National Science Digital Library (NSDL.ORG).
[46] J. Ryan, C. R. McClure, and J. C. Bertot. "Choosing measures to evaluate networked information resources and services: selected issues", in McClure, C.R. and Bertot, J.C. (Eds), Evaluating Networked Information Services: Techniques, Policy, and Issues, Information Today, Medford, NJ, 2001, pp. 111-36.
[47] M. Salamapsis, & K. Diamantaras. Experimental user-centered evaluation of an open hypermedia system and web information-seeking environments. Journal of Digital Information, 2002, vol.2, no.4. (Online) http://jodi.ecs.soton.ac.uk/Articles/v02/i04/Salampasis/
[48] T. Saracevic. Digital library evaluation: toward evolution of concepts. Library Trends, vol. 49, no.2, 2000, pp. 350-369.
[49] T. Saracevic, & L. Covi. Challenges for digital library evaluation. Proceedings of the American Society for Information Science, vol. 37, 2000, pp. 341-350.
[50] M. Sharma, & S. R. Urs. Network Dynamics of Scholarship: A Social Network Analysis of Digital Library Community. Proceeding of the 2nd PhD workshop on Information and Knowledge Management. October 30, 2008, Napa Valley, California, USA. pp: 101-104, 2008.
[51] J. T. Snead, J. C. Bertot, P. T. Jaeger, and C. R. McClure. Developing multi-method, iterative, and user-centered evaluation strategies for digital libraries: functionality, usability, and accessibility", Proceedings of the American Society for Information Science and Technology, vol. 42, no.1, 2005.
[52] K. M. Thompson, C. R. McClure, and P. T. Jaeger., "Evaluating Federal websites: improving e-government for the people", in George, J.F. (Ed.), Computers in Society: Privacy, Ethics, and the Internet, Prentice-Hall, Upper Saddle River, NJ, 2003, pp. 400-412.
[53] G. Tsakonas, S. Kapidakis, C. Papatheodorou. Evaluation of User Interaction in Digital Libraries. Ionian University, 2004. (Online): www.delos.com
[54] University of Rhode Island. Survey of Interest in Creating a Repository of Digital Information at the University of Rhode Island, 2006. ,(Online): http://www.uri.edu/ir/
[55] W. A. K. Wan Dollah. Determining the effectiveness of digital reference services in selected academic libraries in Malaysia. PhD Thesis, University of Malaya, 2008.
[56] H. I. Xie. Users- evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment. Information Processing and Management, vol. 44, 2008, pp. 1346-1373.