Vision-Based Daily Routine Recognition for Healthcare with Transfer Learning

Authors: Bruce X. B. Yu, Yan Liu, Keith C. C. Chan

Abstract:

We propose to record Activities of Daily Living (ADLs) of elderly people using a vision-based system so as to provide better assistive and personalization technologies. Current ADL-related research is based on data collected with help from non-elderly subjects in laboratory environments and the activities performed are predetermined for the sole purpose of data collection. To obtain more realistic datasets for the application, we recorded ADLs for the elderly with data collected from real-world environment involving real elderly subjects. Motivated by the need to collect data for more effective research related to elderly care, we chose to collect data in the room of an elderly person. Specifically, we installed Kinect, a vision-based sensor on the ceiling, to capture the activities that the elderly subject performs in the morning every day. Based on the data, we identified 12 morning activities that the elderly person performs daily. To recognize these activities, we created a HARELCARE framework to investigate into the effectiveness of existing Human Activity Recognition (HAR) algorithms and propose the use of a transfer learning algorithm for HAR. We compared the performance, in terms of accuracy, and training progress. Although the collected dataset is relatively small, the proposed algorithm has a good potential to be applied to all daily routine activities for healthcare purposes such as evidence-based diagnosis and treatment.

Keywords: Daily activity recognition, healthcare, IoT sensors, transfer learning.

References:

[1] U. DESA, "World Population Prospects 2019: Highlights," New York (US): United Nations Department for Economic and Social Affairs, 2019.
[2] J. Gill and M. J. Moore, "The State of aging & health in America 2013," 2013. Available: https://www.statista.com/statistics/207347/causes-ofdeath- among-us-adults-aged-65-by-ethnicity/.
[3] E. C. Nelson, T. Verhagen, and M. L. Noordzij, "Health empowerment through activity trackers: An empirical smart wristband study," Computers in human behavior, vol. 62, pp. 364-374, 2016.
[4] E. Tak, R. Kuiper, A. Chorus, and M. Hopman-Rock, "Prevention of onset and progression of basic ADL disability by physical activity in community dwelling older adults: a meta-analysis," Ageing research reviews, vol. 12, no. 1, pp. 329-338, 2013.
[5] M. Gabel, R. Gilad-Bachrach, E. Renshaw, and A. Schuster, "Full body gait analysis with Kinect," in Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE, 2012: IEEE, pp. 1964-1967.
[6] Y. T. Liao, C.-L. Huang, and S.-C. Hsu, "Slip and fall event detection using Bayesian Belief Network," Pattern recognition, vol. 45, no. 1, pp. 24-32, 2012.
[7] A. Elkholy, M. Hussein, W. Gomaa, D. Damen, and E. Saba, "Efficient and Robust Skeleton-Based Quality Assessment and Abnormality Detection in Human Action Performance," IEEE journal of biomedical and health informatics, 2019.
[8] P. Lukowicz et al., "Recording a complex, multi modal activity data set for context recognition," in Architecture of Computing Systems (ARCS), 2010 23rd International Conference on, 2010: VDE, pp. 1-6.
[9] K. Chapron, P. Lapointe, K. Bouchard, and S. Gaboury, "Highly Accurate Bathroom Activity Recognition using Infrared Proximity Sensors," IEEE Journal of Biomedical and Health Informatics, 2019.
[10] C. Chen, N. Kehtarnavaz, and R. Jafari, "A medication adherence monitoring system for pill bottles based on a wearable inertial sensor," in Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE, 2014: IEEE, pp. 4983-4986.
[11] G. Sprint, D. Cook, D. Weeks, J. Dahmen, and A. La Fleur, "Analyzing sensor-based time series data to track changes in physical activity during inpatient rehabilitation," Sensors, vol. 17, no. 10, p. 2219, 2017.
[12] Q. Ni, A. García Hernando, and I. de la Cruz, "The elderly’s independent living in smart homes: A characterization of activities and sensing infrastructure survey to facilitate services development," Sensors, vol. 15, no. 5, pp. 11312-11362, 2015.
[13] F. Portet, M. Vacher, C. Golanski, C. Roux, and B. Meillon, "Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects," Personal and Ubiquitous Computing, vol. 17, no. 1, pp. 127-144, 2013.
[14] L. Chen, J. Hoey, C. D. Nugent, D. J. Cook, and Z. Yu, "Sensor-based activity recognition," IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 790- 808, 2012.
[15] F. Palumbo, J. Ullberg, A. Štimec, F. Furfari, L. Karlsson, and S. Coradeschi, "Sensor network infrastructure for a home care monitoring system," Sensors, vol. 14, no. 3, pp. 3833-3860, 2014.
[16] H. Alemdar and C. Ersoy, "Wireless sensor networks for healthcare: A survey," Computer networks, vol. 54, no. 15, pp. 2688-2710, 2010.
[17] J. K. Aggarwal and L. Xia, "Human activity recognition from 3d data: A review," Pattern Recognition Letters, vol. 48, pp. 70-80, 2014.
[18] S. C. Mukhopadhyay, "Wearable sensors for human activity monitoring: A review," IEEE sensors journal, vol. 15, no. 3, pp. 1321-1330, 2015.
[19] O. D. Lara and M. A. Labrador, "A survey on human activity recognition using wearable sensors," IEEE Communications Surveys and Tutorials, vol. 15, no. 3, pp. 1192-1209, 2013.
[20] A. Wickramasinghe, R. L. S. Torres, and D. C. Ranasinghe, "Recognition of falls using dense sensing in an ambient assisted living environment," Pervasive and mobile computing, vol. 34, pp. 14-24, 2017.
[21] W. Wang, A. X. Liu, M. Shahzad, K. Ling, and S. Lu, "Understanding and modeling of wifi signal based human activity recognition," in Proceedings of the 21st annual international conference on mobile computing and networking, 2015: ACM, pp. 65-76.
[22] H. Sagha et al., "Benchmarking classification techniques using the Opportunity human activity dataset," in Systems, Man, and Cybernetics (SMC), 2011 IEEE International Conference on, 2011: IEEE, pp. 36-40.
[23] F. Ofli, R. Chaudhry, G. Kurillo, R. Vidal, and R. Bajcsy, "Berkeley MHAD: A comprehensive multimodal human action database," in Applications of Computer Vision (WACV), 2013 IEEE Workshop on, 2013: IEEE, pp. 53-60.
[24] C. Chen, R. Jafari, and N. Kehtarnavaz, "A survey of depth and inertial sensor fusion for human action recognition," Multimedia Tools and Applications, vol. 76, no. 3, pp. 4405-4425, 2017.
[25] F. J. Ordóñez and D. Roggen, "Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition," Sensors, vol. 16, no. 1, p. 115, 2016.
[26] J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras," in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2012: IEEE, pp. 1290- 1297.
[27] C. Liu, Y. Hu, Y. Li, S. Song, and J. Liu, "PKU-MMD: A Large Scale Benchmark for Skeleton-Based Human Action Understanding," in Proceedings of the Workshop on Visual Analysis in Smart and Connected Communities, 2017: ACM, pp. 1-8.
[28] J. Liu, A. Shahroudy, M. L. Perez, G. Wang, L.-Y. Duan, and A. K. Chichung, "NTU RGB+ D 120: A Large-Scale Benchmark for 3D Human Activity Understanding," IEEE transactions on pattern analysis and machine intelligence, 2019.
[29] T. Van Kasteren, A. Noulas, G. Englebienne, and B. Kröse, "Accurate activity recognition in a home setting," in Proceedings of the 10th international conference on Ubiquitous computing, 2008: ACM, pp. 1-9.
[30] J. A. Stork, L. Spinello, J. Silva, and K. O. Arras, "Audio-based human activity recognition using non-markovian ensemble voting," in RO-MAN, 2012 IEEE, 2012: IEEE, pp. 509-514.
[31] L. Guo, L. Wang, J. Liu, W. Zhou, and B. Lu, "HuAc: Human Activity Recognition Using Crowdsourced WiFi Signals and Skeleton Data," Wireless Communications and Mobile Computing, vol. 2018, 2018.
[32] A. Reiss and D. Stricker, "Creating and benchmarking a new dataset for physical activity monitoring," in Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments, 2012: ACM, p. 40.
[33] S. Katz, "Assessing self‐maintenance: activities of daily living, mobility, and instrumental activities of daily living," Journal of the American Geriatrics Society, vol. 31, no. 12, pp. 721-727, 1983.
[34] A. Jalal, S. Kamal, and D. Kim, "A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments," Sensors, vol. 14, no. 7, pp. 11735-11759, 2014.
[35] D. M. Gavrila and L. S. Davis, "Towards 3-d model-based tracking and recognition of human movement: a multi-view approach," in International workshop on automatic face-and gesture-recognition, 1995: Citeseer, pp. 272-277.
[36] N. Oliver, E. Horvitz, and A. Garg, "Layered representations for human activity recognition," in Proceedings. Fourth IEEE International Conference on Multimodal Interfaces, 2002: IEEE, pp. 3-8.
[37] R. Lublinerman, N. Ozay, D. Zarpalas, and O. Camps, "Activity recognition from silhouettes using linear systems and model (in) validation techniques," in 18th International Conference on Pattern Recognition (ICPR'06), 2006, vol. 1: IEEE, pp. 347-350.
[38] J. Wang, Y. Chen, S. Hao, X. Peng, and L. Hu, "Deep Learning for Sensor-based Activity Recognition: A Survey," arXiv preprint arXiv:1707.03502, 2017.
[39] "Kinect tools and resources." Available: https://developer.microsoft.com/en-us/windows/kinect/tools.
[40] F. Lv and R. Nevatia, "Recognition and segmentation of 3-d human action using hmm and multi-class adaboost," in European conference on computer vision, 2006: Springer, pp. 359-372.
[41] J. Liu, A. Shahroudy, D. Xu, and G. Wang, "Spatio-temporal lstm with trust gates for 3d human action recognition," in European Conference on Computer Vision, 2016: Springer, pp. 816-833.
[42] S. Song, C. Lan, J. Xing, W. Zeng, and J. Liu, "An End-to-End Spatio- Temporal Attention Model for Human Action Recognition from Skeleton Data," in AAAI, 2017, pp. 4263-4270.
[43] J. Liu, G. Wang, P. Hu, L.-Y. Duan, and A. C. Kot, "Global contextaware attention lstm networks for 3d action recognition," in CVPR, 2017.
[44] P. Zhang, C. Lan, J. Xing, W. Zeng, J. Xue, and N. Zheng, "View adaptive recurrent neural networks for high performance human action recognition from skeleton data," arXiv, no. Mar, 2017.
[45] M. Liu, H. Liu, and C. Chen, "Enhanced skeleton visualization for view invariant human action recognition," Pattern Recognition, vol. 68, pp. 346-362, 2017.
[46] P. Wei, Y. Zhao, N. Zheng, and S.-C. Zhu, "Modeling 4D human-object interactions for joint event segmentation, recognition, and object localization," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 6, pp. 1165-1179, 2017.
[47] S. Althloothi, M. H. Mahoor, X. Zhang, and R. M. Voyles, "Human activity recognition using multi-features and multiple kernel learning," Pattern recognition, vol. 47, no. 5, pp. 1800-1812, 2014.
[48] J. Liu, A. Shahroudy, D. Xu, A. K. Chichung, and G. Wang, "Skeleton- Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[49] S. Yan, Y. Xiong, and D. Lin, "Spatial temporal graph convolutional networks for skeleton-based action recognition," arXiv preprint arXiv:1801.07455, 2018.
[50] J. Dean. "Building Intelligent Systems withLarge Scale Deep Learning." Available: https://zh.scribd.com/document/355752799/Jeff-Dean-s- Lecture-for-YC-AI.
[51] S. Ramasamy Ramamurthy and N. Roy, "Recent trends in machine learning for human activity recognition—A survey," Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 8, no. 4, p. e1254, 2018.
[52] K. Weiss, T. M. Khoshgoftaar, and D. Wang, "A survey of transfer learning," Journal of Big data, vol. 3, no. 1, p. 9, 2016.
[53] S. V. Stehman, "Selecting and interpreting measures of thematic classification accuracy," Remote sensing of Environment, vol. 62, no. 1, pp. 77-89, 1997.
[54] G. James, D. Witten, T. Hastie, and R. Tibshirani, An introduction to statistical learning. Springer, 2013.