Application of Scientific Metrics to Evaluate Academic Reputation in Different Research Areas

Authors: Cristiano R. Cervi, Renata Galante, José Palazzo M. de Oliveira

Abstract:

In this paper, we address the problem of identifying academic reputation of researchers using scientific metrics in different research areas. Due to the characteristics of each area, researchers can present different behaviors. In previous work, we define Rep-Index that makes use of a profile template to individually identify the reputation of researchers. The Rep-Index is comprehensive and adaptive because involves hole trajectory of the researcher built throughout his career and can be used in different areas and in different contexts. Now, we compare our metric (Rep-Index) with the h-index and the g-index through experiments with researchers in the fields of Economics, Dentistry and Computer Science. We analyze the trajectory of 830 Brazilian researchers from the National Council of Technological and Scientific Development (CNPq), which receive grants research productivity. The grants are aimed at productivity researchers that stand out among their peers, enhancing their scientific normative criteria established by CNPq. Of the 830 researchers, 210 are in the area of Economics, 216 of Dentistry e 404 of Computer Science. The experiments show that our metric is strongly correlated with h-index, g-index and CNPq ranking. We also show good results for our hypothesis that our metric can be used to evaluate research in several areas. We apply our metric (Rep-Index) to compare the behavior of researchers in relation to their h-index and g-index through extensive experiments. The experiments showed that our metric is strongly correlated with h-index, g-index and CNPq ranking.

Keywords: Researcher reputation, profile model, scientific metrics.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1088532

References:

[1] J. E. Hirsch, “An Index to Quantify an Individual's Scientific Research Output”, Proceedings of the National Academy of Science, vol.102, no.46, 2005, pp. 16569-16572.
[2] L. Egghe, “Theory and Practise of the g-index”, Scientometrics, vol.69, no.1, 2006, pp. 131-152.
[3] B. Jin, “The AR-Index: complementing the h-index”, International Society for Scientometrics and Informetrics (ISSI Newsletter), vol.3 no.1, 2007, pp. 6.
[4] B. Jin, L. Liming, R. Rousseau, L. Egghe, “The R- and AR-indices: complementing the h-index, Chinese Science Bulletin, vol.52, no.6, 2007, pp. 855-863.
[5] C. Zhang, “The e-index, complementing the h-index for excess citations”, PLoSONE, vol.4, no.5, 2009, e5429.
[6] C. R. Cervi, R. Galante, J. P. M. Oliveira, “An adaptive approach for identifying reputation of researchers”, Proceedings of the twelve International Conference on WWW/Internet, Madrid, Spain, 2012.
[7] C. R. Cervi, R. Galante, J. P. M. Oliveira, “Comparing the reputation of researchers using a profile model and scientific metrics”, unpublished.
[8] S. Alonso, F. J. Cabrerizo, E. Herrera-Viedma, F. Herrera, “H-index: A review focused in its variants, computation and standardization for different scientific fields”, Journal of Informetrics, vol.3, no.4, 2009, pp.273-289.
[9] L. Bornmann, R. Mutz, H. D. Daniel, “Are there better indices for evaluation purposes than the h-index? A comparison of nine different variants of the h-index using data from biomedicine”, Journal of the American Society for Information Science and Technology, vol.59, no.5, 2008, pp. 830-837.
[10] M. V. Dodson, “Citation analysis: Maintenance of h-index and use of e-index”, Biochemical and Biophysical Research Communications 387, 2009, pp. 625-626.
[11] R. S. J. Tol, “The h-index and its alternatives: An application to the 100 most prolific economists”, Scientometrics, vol.80, no.2, 2009, pp. 317-324.
[12] J. Mingers, F. Macri, D. Petrovici, “Using the h-index to measure the quality of journals in the field of business and management”, Information Processing and Management: an International Journal, vol.48, no.2, 2012, pp. 234-241.
[13] S. Papavlasopoulos, M. Poulos, N. Korfiatis, G. Bokos, “A non-linear index to evaluate a journal's scientific impact”, Information Sciences: an International Journal, vol.180, no.11, 2010, pp. 2156-2175.
[14] D. Carmel, V. Josifovski, Y. Maarek, “User modeling for web applications”, Proceedings of the fourth ACM International Conference on Web Search and Data Mining, Hong Kong, China, 2011.
[15] K. Schoefegger, “A user modeling approach to support knowledge work in socio-computational systems”, Proceedings of the 19th Conference on User Modeling, Adaptation and Personalization, Girona, Spain, 2011.
[16] Z. Saaya, B. Smyth, M. Coyle, P. Briggs, “Recognising and recommending context in social web search”, Proceedings of the 19th Conference on User Modeling, Adaptation and Personalization, Girona, Spain, 2011.
[17] I. Gasparini, A. M. Pernas, M. S. Pimenta, J. P. M. Oliveira, A. Kemczinski, G. G. H. Cavalheiro, “m-AdaptWeb: an adaptive e-learning environment facing mobility - adaptation and recomendation processes based on context. Proceedings of fourth International Conference on Computer Supported Education, Porto, Portugal, 2012, pp. 395-400.
[18] J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, Z. Su, “ArnetMiner - extraction and mining of academic social networks”, Proceeding of the 14th Conference on Knowledge Discovery and Data Mining, Las Vegas, USA, 2008.
[19] C. Chen, I. Song, X. Yuan, J. Zhang, “The thematic and citation landscape of data and knowledge engineering”, Data & Knowledge Engineering, vol.67, no.2, 2008, pp. 234-259.
[20] G. R. Lopes, R. da Silva, M. M. Moro, J. P. M. Oliveira, “Scientific collaboration in research networks: a quantification method by using Ginicoefficient”, International Journal of Computer Science & Applications, vol.9, 2012, pp. 15-31.
[21] R. Punnarut, G. Sriharee, “A researcher expertise search system using ontology-based data mining”, Proceedings of the 7th Asia-Pacific Conference on Conceptual Modelling, Brisbane, Australia, 2010.
[22] Y. Ding, E. Yan, A. Frazho, J. Caverlee, “PageRank for ranking authors in co-citation networks”, Journal of the American Society for Information Science and Technology, vol.60, no.11, 2009, pp. 2229-2243.
[23] M. Krapivin, M. Marchese, F. Casati, “Exploring and understanding scientific metrics in citation networks”, Complex Sciences, vol.5, 2009, pp. 1550-1563.
[24] S. Loh, R. L. Granada, D. Litchnow, L. K. Wives, J. P. M. Oliveira, F. Lorenzi, “Using scientific publications to identify people”, Lecture Notes in Business Information Processing, vol.45, 2010, pp. 229-241.
[25] E. A. Oliveira, E. A. Colosimo, D. R. Martelli, I. G. Quirino, M. C. L. Oliveira, L. S. Lima, A. C. Simões e Silva, H. Martelli-Júnior, “Comparison of Brazilian researchers in clinical medicine: are criteria for ranking well-adjusted?”, Scientometrics vol.90, no.2, 2012, pp. 429-443.
[26] F. R. Spilki, “Perfil dos bolsistas de produtividade do Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) na área de medicina veterinária”, Pesquisa Veterinária Brasileira, vol.33, n.2, 2013, pp. 205-213.
[27] S. Alonso, F. J. Cabrerizo, E. Herrera-Viedma, F. Herrera, “Hg-index: A new index to characterize the scientific output of researchers based on the h- and g-indices”, Scientometrics, vol.82, no.2, 2010, pp. 391-400.
[28] X. B. Yan, L. Zhai, W. G. Fan, “C-index: a weighted network node centrality measure for collaboration competence”, Journal of Informetrics, vol.7, no.1, 2013, pp. 223-239.
[29] L. Zhai, X. Yan, B. Zhu, “The H l - index: improvement of H-index based on quality of citing papers”, Scientometrics, 2013, pp. 1-11.
[30] C. T. Zhang, “The h’-Index, effectively improving the h-index based on the citation distribution”, PLoS ONE, vol.8, no4, 2013, e59912.
[31] A. Harzing, R. Van Der Wal, “A Google Scholar h-index for journals: An alternative metric to measure journal impact in economics and business”, Journal of the American Society for Information Science and Technology, vol.60, no.1, 2009, pp. 41-46.
[32] C. T. Zhang, “A novel triangle mapping technique to study the h-index based citation distribution”, Scientific Report, vol.3, no.1023, 2013, pp. 1-5.
[33] A. M. Abbas, “Bounds and Inequalities Relating h-index, g-index, e-index and Generalized Impact Factor: An Improvement over Existing Models”, PLoS ONE 7: e33699, 2012.
[34] F. Y. Ye, “A unification of three models for the h-index”, Journal of the American Society for Information Science and Technology, vol.62, no.1, 2011, pp. 205-207.
[35] R. Rousseau, B. H. Jin, “The age-dependent h-type AR(2)-index: basic properties and a case study”, Journal of the American Society for Information Science and Technology, vol.59, no.14, 2008, pp. 2305-2311.
[36] M. Schreiber, C. C. Malesios, S. Psarakis, “Exploratory factor analysis for the Hirsch index, 17 h-type variants, and some traditional bibliometric indicators”, Journal of Informetrics, vol.6, no.3, 2012, pp. 347-358.
[37] R. Rousseau, C. Garcia-Zorita, E. Sanz-Casado, “The h-bubble”, Journal of Informetrics, vol.7, no.2, 2013, pp. 294-300.
[38] F. Y. Ye, R. Rousseau, “Probing the h-core: an investigation of the tail-core ratio for rank distributions”, Scientometrics, vol.84, no.2, 1010, pp. 431-439.
[39] J. Wainer, P. Vieira, “Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists”, Scientometrics, vol.96, no.2, 2013, pp.395-410.
[40] G. R. Lopes, M. M. Moro, R. da Silva, E. M. Barbosa, J. P. M. Oliveira, “Ranking Strategy for Graduate Programs Evaluation”, Proceedings of the 7th International Conference on Information Technology and Application, Sydney, Australia, 2011.
[41] M. Franceschet, A. Costantini, “The First Italian Research Assessment Exercise: A bibliometric perspective”, Journal of Informetrics, vol.5, no.2, 2011, pp. 275-291.
[42] L. Waltman, N. Van Eck, T. Van Leeuwen, M. Visser, A. Van Raan, “On the correlation between bibliometric indicators and peer review: Reply to Opthof and Leydesdor”, Scientometrics, vol.88, no.3, 2011, pp. 1017-1022.
[43] J. Li, M. Sanderson, P. Willett, M. Norris, C. Oppenheim, “Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments”, Journal of Informetrics, vol.4, no.4, 2010, pp. 554-563.