Insight-Based Evaluation of a Map-based Dashboard

Authors: Anna Fredriksson Häägg, Charlotte Weil, Niklas Rönnberg

Abstract:

Map-based dashboards are used for data exploration every day. The present study used an insight-based methodology for evaluating a map-based dashboard that presents research findings of water management and ecosystem services in the Amazon. In addition to analyzing the insights gained from using the dashboard, the evaluation method was compared to standardized questionnaires and task-based evaluations. The result suggests that the dashboard enabled the participants to gain domain-relevant, complex insights regarding the topic presented. Furthermore, the insight-based analysis highlighted unexpected insights and hypotheses regarding causes and potential adaptation strategies for remediation. Although time- and resource-consuming, the insight-based methodology was shown to have the potential of thoroughly analyzing how end users can utilize map-based dashboards for data exploration and decision making. Finally, the insight-based methodology is argued to evaluate tools in scenarios more similar to real-life usage, compared to task-based evaluation methods.

Keywords: Visual analytics, dashboard, insight-based evaluation, geographic visualization.

References:

[1] I. D. Bishop, C. J. Pettit, F. Sheth, and S. Sharma, “Evaluation of data visualisation options for land-use policy and decision making in response to climate change,” Environment and Planning B: Planning and Design, vol. 40, no. 2, pp. 213–233, 2013.
[2] R. Moss, P. L. Scarlett, M. Kenney, H. Kunreuther, R. Lempert, J. Manning, B. Williams, J. Boyd, E. Cloyd, L. Kaatz et al., “Decision support: Connecting science, risk perception, and decisions,” in Proc. Climate change impacts in the United States: The third national climate assessment. US Global Change Research Program, 2014, pp. 620–647.
[3] A. Bohman, T.-S. Neset, T. Opach, and J. K. Rød, “Decision support for adaptive action – assessing the potential of geographic visualization,” Journal of Environmental Planning and Management, vol. 58, pp. 1–19, 05 2014.
[4] M. Allio, “Strategic dashboards: Designing and deploying them to improve implementation,” Strategy and Leadership, vol. 40, 08 2012.
[5] NOAA Office for Coastal Management, Ocean Reports, Accessed February 4, 2021.
[Online]. Available: https://marinecadastre.gov/oceanreports/
[6] World Resources Institute, Resource Watch, Accessed February 4, 2021. (Online). Available: https://resourcewatch.org/dashboards
[7] the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU), COVID-19 Dashboard, Accessed February 1, 2022. (Online). Available: https://coronavirus.jhu.edu/map.html
[8] T. Isenberg, P. Isenberg, J. Chen, M. Sedlmair, and T. M¨oller, “A systematic review on the practice of evaluating visualization,” IEEE Transactions on Visualization and Computer Graphics, vol. 19, no. 12, pp. 2818–2827, 2013.
[9] C. North, “Toward measuring visualization insight,” IEEE Computer Graphics and Applications, vol. 26, no. 3, pp. 6–9, 2006.
[10] G. Sedrakyan, E. Mannens, and K. Verbert, “Guiding the choice of learning dashboard visualizations: Linking dashboard design and data visualization concepts,” Journal of Computer Languages, vol. 50, pp. 19–38, 2019.
[11] A. Sarikaya, M. Correll, L. Bartram, M. Tory, and D. Fisher, “What do we talk about when we talk about dashboards?” IEEE Transactions on Visualization and Computer Graphics, pp. 136–142, 08 2018.
[12] S. R. Sheppard, A. Shaw, D. Flanders, S. Burch, A. Wiek, J. Carmichael, J. Robinson, and S. Cohen, “Future visioning of local climate change: A framework for community engagement and planning with scenarios and visualisation,” Futures, vol. 43, no. 4, pp. 400–412, 2011, special Issue: Community Engagement for Sustainable Urban Futures.
[13] S. Grainger, F. Mao, and W. Buytaert, “Environmental data visualisation for non-scientific contexts: Literature review and design framework,” Environmental Modelling & Software, vol. 85, pp. 299–318, 2016.
[14] B. Shneiderman, “The eyes have it: a task by data type taxonomy for information visualizations,” in Proc. 1996 IEEE Symposium on Visual Languages. Boulder, CO, USA: IEEE, 1996, pp. 336–343.
[15] S. W. Marvel, J. S. House, M. Wheeler, K. Song, Y.-H. Zhou, F. A. Wright, W. A. Chiu, I. Rusyn, A. Motsinger-Reif, and D. M. Reif, “The covid-19 pandemic vulnerability index (pvi) dashboard: Monitoring county-level vulnerability using visualization, statistical modeling, and machine learning,” Environmental Health Perspectives, vol. 129, no. 1, p. 017701, 2021.
[16] W. W. Eckerson, Types of Performance Dashboards. John Wiley & Sons, Ltd, 2015, ch. 6, pp. 101–121.
[17] R. E. Roth, A. C¸ ¨oltekin, L. Delazari, H. F. Filho, A. Griffin, A. Hall, J. Korpi, I. Lokka, A. Mendonc¸a, K. Ooms, and C. P. van Elzakker, “User studies in cartography: opportunities for empirical research on interactive maps and visualizations,” International Journal of Cartography, vol. 3, pp. 61–89, 2017.
[18] VisGap’20 - Workshop, The gap between visualization research and visualization software development, May 25, 2020. Accessed February 4, 2021, workshop in conjunction with EGEV2020, Norrk¨oping, Sweden. (Online). Available: https://visgap.gitlab.io/visgap20/
[19] S. Saha, S. Shekhar, S. Sadhukhan, and P. Das, “An analytics dashboard visualization for flood decision support system,” Journal of Visualization, vol. 21, no. 2, pp. 295–307, 2018.
[20] J. A´ lvarez-Francoso, J. Ojeda-Zu´jar, P. D´ıaz-Cuevas, E. Guisado-Pintado, J. Camarillo-Naranjo, A. Prieto-Campos, and P. Fraile-Jurado, “A specialized geoviewer and dashboard for beach erosion rates visualization and exploration,” Journal of Coastal Research, vol. 95, no. SI, pp. 1006–1010, 2020.
[21] K. Vincent, R. Roth, S. Moore, Q. Huang, N. Lally, C. Sack, E. Nost, and H. Rosenfeld, “Improving spatial decision making using interactive maps: An empirical study on interface complexity and decision complexity in the north american hazardous waste trade,” Environment and Planning B: Urban Analytics and City Science, vol. 46, p. 239980831876412, 04 2018.
[22] S. Yamamoto and H. Mori, Human Interface and the Management of Information. Designing Information: Thematic Area, HIMI 202, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part I. Springer, 2020, vol. 12184.
[23] R. Roth, “Interactive maps: What we know and what we need to know,” Journal of Spatial Information Science, vol. 6, pp. 59–115, 06 2013.
[24] C. Forsell, “Evaluation in information visualization: Heuristic evaluation,” in Proc. 2012 16th International Conference on Information Visualisation. Montpellier, France: IEEE, 2012, pp. 136–142.
[25] S. Carpendale, “Evaluating information visualizations,” in Information Visualization, A. Kerren, J. Stasko, J. Fekete, and C. North, Eds. Berlin, Heidelberg: Springer, 2008, vol. 4950, pp. 19–45.
[26] H. Lam, E. Bertini, P. Isenberg, C. Plaisant, and S. Carpendale, “Empirical studies in information visualization: Seven scenarios,” IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 9, pp. 1520–1536, 2012.
[27] M. Tory and T. M¨oller, “Evaluating visualizations: do expert reviews work?” IEEE Computer Graphics and Applications, vol. 25, no. 5, pp. 8–11, 2005.
[28] J. Stasko, “Value-driven evaluation of visualizations,” in Proc. of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization. Paris, France: Association for Computing Machinery, 2014, pp. 46–53.
[29] P. Saraiya, C. North, and K. Duca, “An insight-based methodology for evaluating bioinformatics visualizations,” IEEE Transactions on Visualization and Computer Graphics, vol. 11, no. 4, pp. 443–456, 2005.
[30] A. Ledesma, N. Bidargaddi, J. Strobel, G. Schrader, H. Nieminen, I. Korhonen, and M. Ermes, “Health timeline: an insight-based study of a timeline visualization of clinical data,” BMC Medical Informatics and Decision Making, vol. 19, pp. 1–14, 12 2019.
[31] A. Ledesma, H. Nieminen, P. Valve, M. Ermes, H. Jimison, and M. Pavel, “The shape of health: A comparison of five alternative ways of visualizing personal health and wellbeing,” in Proc. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Milan, Italy: IEEE, 2015, pp. 7638–7641.
[32] P. Saraiya, C. North, Vy Lam, and K. A. Duca, “An insight-based longitudinal study of visual analytics,” IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 6, pp. 1511–1522, 2006.
[33] The Natural Capital Project, PRO Agua viewer, Accessed February 4, 2021. (Online). Available: http://viz.naturalcapitalproject.org/pro-agua/index.html
[34] J. Brooke, “SUS - A quick and dirty usability scale,” Usability evaluation in industry, vol. 189, no. 194, pp. 4–7, 1996.