Unsupervised Echocardiogram View Detection via Autoencoder-Based Representation Learning

Authors: Andrea Treviño Gavito, Diego Klabjan, Sanjiv J. Shah

Abstract:

Echocardiograms serve as pivotal resources for clinicians in diagnosing cardiac conditions, offering non-invasive insights into a heart’s structure and function. When echocardiographic studies are conducted, no standardized labeling of the acquired views is performed. Employing machine learning algorithms for automated echocardiogram view detection has emerged as a promising solution to enhance efficiency in echocardiogram use for diagnosis. However, existing approaches predominantly rely on supervised learning, necessitating labor-intensive expert labeling. In this paper, we present a fully unsupervised echocardiographic view detection framework that leverages convolutional autoencoders to obtain lower dimensional representations and the K-means algorithm for clustering them into view-related groups. Our approach focuses on discriminative patches from echocardiographic frames. Additionally, we propose a trainable inverse average layer to optimize decoding of average operations. By integrating both public and proprietary datasets, we obtain a marked improvement in model performance when compared to utilizing a proprietary dataset alone. Our experiments show boosts of 15.5% in accuracy and 9.0% in the F-1 score for frame-based clustering, and 25.9% in accuracy and 19.8% in the F-1 score for view-based clustering. Our research highlights the potential of unsupervised learning methodologies and the utilization of open-sourced data in addressing the complexities of echocardiogram interpretation, paving the way for more accurate and efficient cardiac diagnoses.

Keywords: Artificial intelligence, echocardiographic view detection, echocardiography, machine learning, self-supervised representation learning, unsupervised learning.

References:

[1] A.-R. Ali, A. E. Samir, and P. Guo. Self-supervised learning for accurate liver view classification in ultrasound images with minimal labeled data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023.
[2] S. Azizi, F. Imani, B. Zhuang, A. Tahmasebi, J. T. Kwak, S. Xu, N. Uniyal, B. Turkbey, P. Choyke, P. Pinto, B. Wood, M. Moradi, P. Mousavi, and P. Abolmaesumi. Ultrasound-based detection of prostate cancer using automatic feature selection with deep belief networks. In Medical Image Computing and Computer-Assisted Intervention, 2015.
[3] S. Basu, S. Singla, M. Gupta, P. Rana, P. Gupta, and C. Arora. Unsupervised Contrastive Learning of Image Representations from Ultrasound Videos with Hard Negative Mining. In Medical Image Computing and Computer Assisted Intervention, 2022.
[4] G. Carneiro and J. C. Nascimento. Multiple dynamic models for tracking the left ventricle of the heart from ultrasound data using particle filters and deep learning architectures. In IEEE Conference on Computer Vision and Pattern Recognition, 2010.
[5] G. Carneiro and J. C. Nascimento. Combining multiple dynamic models and deep learning architectures for tracking the left ventricle endocardium in ultrasound data. IEEE Transactions on Pattern Analysis Machine Intelligence, 2013.
[6] G. Carneiro, J. C. Nascimento, and A. Freitas. The segmentation of the left ventricle of the heart from ultrasound data using deep learning architectures and derivative-based search methods. IEEE Transactions on Image Processing, 2012.
[7] H. Chen, Q. Dou, D. Ni, J.-Z. Cheng, J. Qin, S. Li, and P.-A. Heng. Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks. In Medical Image Computing and Computer-Assisted Intervention, 2015.
[8] Y. Chen, C. Zhang, L. Liu, C. Feng, C. Dong, Y. Luo, and X. Wan. USCL: Pretraining deep ultrasound image diagnosis model through video contrastive representation learning. In Medical Image Computing and Computer Assisted Intervention, 2021.
[9] J.-Z. Cheng, D. Ni, Y.-H. Chou, J. Qin, C.-M. Tiu, Y.-C. Chang, C.-S. Huang, D. Shen, and C.-M. Chen. Computer-aided diagnosis with deep learning architecture: Applications to breast lesions in us images and pulmonary nodules in CT scans. Scientific Reports, 2016.
[10] R. Delaunay, Y. Hu, and T. Vercauteren. An unsupervised approach to ultrasound elastography with end-to-end strain regularisation. In Medical Image Computing and Computer Assisted Intervention, 2020.
[11] R. Delaunay, Y. Hu, and T. Vercauteren. An unsupervised learning approach to ultrasound strain elastography with spatio-temporal consistency. Physics in Medicine & Biology, 2021.
[12] R. C. Deo, J. Zhang, L. A. Hallock, S. Gajjala, L. Nelson, E. Fan, M. A. Aras, C. Jordan, K. E. Fleischmann, M. Melisko, A. Qasim, S. J. Shah, and R. Bajcsy. An end-to-end computer vision pipeline for automated cardiac function assessment by echocardiography. ArXiv, abs/1706.07342, 2017.
[13] F. Dezaki, C. Luong, T. Ginsberg, R. Rohling, K. Gin, P. Abolmaesumi, and T. Tsang. Echo-SyncNet: Self-supervised cardiac view synchronization in echocardiography. IEEE Transactions on Medical Imaging, 2021.
[14] R. Droste, Y. Cai, H. Sharma, P. Chatelain, L. Drukker, A. Papageorghiou, and J. Noble. Ultrasound image representation learning by modeling sonographer visual attention. In Information Processing in Medical Imaging, 2019.
[15] D. L. Ferreira, Z. Salaymang, and R. Arnaout. Self-supervised learning for label-free segmentation in cardiac ultrasound. ArXiv, abs/2210.04979, 2022.
[16] Z. Fu, J. Jiao, R. Yasrab, L. Drukker, A. T. Papageorghiou, and J. A. Noble. Anatomy-aware contrastive representation learning for fetal ultrasound. European Conference on Computer Vision, 2022.
[17] X. Gao, W. Li, M. Loomes, and L. Wang. A fused deep learning architecture for viewpoint classification of echocardiography. Information Fusion, 2017.
[18] Y. Gao, Y. Zhu, B. Liu, Y. Hu, G. Yu, and Y. Guo. Automated recognition of ultrasound cardiac views based on deep learning with graph constraint. Diagnostics, 2021.
[19] X. Guo, X. Liu, E. Zhu, and J. Yin. Deep clustering with convolutional autoencoders. In Neural Information Processing, 2017.
[20] G. Holste, E. K. Oikonomou, B. J. Mortazavi, Z. Wang, and R. Khera. Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis. ArXiv, abs/2207.11581, 2023.
[21] S.-Y. Hu, S. Wang, W.-H. Weng, J. Wang, X. Wang, A. Ozturk, Q. Li, V. Kumar, and A. E. Samir. Self-supervised pretraining with DICOM metadata in ultrasound imaging. In Machine Learning for Healthcare, 2020.
[22] Y. Hu, T. M. Sutter, E. Ozkan, and J. E. Vogt. Self-supervised learning to predict ejection fraction using motion-mode images. In International Conference on Learning Representations, Workshop on Machine Learning & Global Health, 2023.
[23] A. Jaumard-Hakoun, K. Xu, P. Roussel-Ragot, G. Dreyfus, and B. Denby. Tongue contour extraction from ultrasound images based on deep neural network. ArXiv, abs/1605.05912, 2016.
[24] J. Jiao, Y. Cai, M. Alsharid, L. Drukker, A. T. Papageorghiou, and J. A. Noble. Self-supervised contrastive video-speech representation learning for ultrasound. In Medical Image Computing and Computer-Assisted Intervention, 2020.
[25] J. Jiao, R. Droste, L. Drukker, A. Papageorghiou, and J. Noble. Self-supervised representation learning for ultrasound video. In IEEE International Symposium on Biomedical Imaging, 2020.
[26] H. Khamis, G. Zurakhov, V. Azar, A. Raz, Z. Friedman, and D. Adam. Automatic apical view classification of echocardiograms using a discriminative learning dictionary. Medical Image Analysis, 2017.
[27] K. Kusunose. Steps to use artificial intelligence in echocardiography. Journal of Echocardiography, 2020.
[28] K. Kusunose, A. Haga, M. Inoue, D. Fukuda, H. Yamada, and M. Sata. Clinically feasible and accurate view classification of echocardiographic images using deep learning. Biomolecules, 2020.
[29] S. Leclerc, E. Smistad, J. Pedrosa, A. Østvik, F. Cervenansky, F. Espinosa, T. Espeland, E. A. R. Berg, P.-M. Jodoin, T. Grenier, C. Lartizien, J. D’hooge, L. Løvstakken, and O. Bernard. Deep learning for segmentation using an open large-scale dataset in 2D echocardiography. IEEE Transactions on Medical Imaging, 2019.
[30] Z. Liao, M. H. Jafari, H. Girgis, K. Gin, R. Rohling, P. Abolmaesumi, and T. Tsang. Echocardiography view classification using quality transfer star generative adversarial networks. In Medical Image Computing and Computer Assisted Intervention, 2019.
[31] X. Liu, Y. Fan, S. Li, M. Chen, M. Li, W. K. Hau, H. Zhang, L. Xu, and A. P.-W. Lee. Deep learning-based automated left ventricular ejection fraction assessment using 2-D echocardiography. American Journal of Physiology-Heart and Circulatory Physiology, 2021.
[32] J. Lu and W. Liu. Unsupervised super-resolution framework for medical ultrasound images using dilated convolutional neural networks. In IEEE International Conference on Image, Vision and Computing, 2018.
[33] A. Madani, R. Arnaout, M. Mofrad, and R. Arnaout. Fast and accurate view classification of echocardiograms using deep learning. Digital Medicine, 2018.
[34] A. Madani, J. Ong, A. Tibrewal, and M. Mofrad. Deep echocardiography: Data-efficient supervised and semi-supervised deep learning towards automated diagnosis of cardiac disease. npj Digital Medicine, 2018.
[35] R.-M. Mench´on-Lara and J. L. Sancho-G´omez. Fully automatic segmentation of ultrasound common carotid artery images based on machine learning. Neurocomputing, 2015.
[36] F. Milletari, S.-A. Ahmadi, C. Kroll, C. Hennersperger, F. Tombari, A. Shah, A. Plate, K. Boetzel, and N. Navab. Robust segmentation of various anatomies in 3d ultrasound using hough forests and learned data representations. In Medical Image Computing and Computer-Assisted Intervention, 2015.
[37] D. Ouyang, B. He, A. Ghorbani, N. Yuan, J. Ebinger, C. P. Langlotz, P. A. Heidenreich, R. A. Harrington, D. Liang, E. A. Ashley, and J. Y. Zou. Video-based AI for beat-to-beat assessment of cardiac function. Nature, 2020.
[38] K. T. Shahid and I. Schizas. Unsupervised mitral valve tracking for disease detection in echocardiogram videos. Journal of Imaging, 2020.
[39] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[40] J. Tromp, P. Seekings, C.-L. Hung, M. Iversen, M. Frost, W. Ouwerkerk, Z. Jiang, F. Eisenhaber, R. Goh, H. Zhao, W. Huang, L.-H. Ling, D. Sim, P. Cozzone, A. Richards, H. Lee, S. Solomon, C. Lam, and J. Ezekowitz. Automated interpretation of systolic and diastolic function on the echocardiogram: a multicohort study. The Lancet Digital Health, 2022.
[41] H. Vaseli, Z. Liao, A. H. Abdi, H. Girgis, D. Behnami, C. Luong, F. T. Dezaki, N. Dhungel, R. Rohling, K. Gin, P. Abolmaesumi, and T. Tsang. Designing lightweight deep learning models for echocardiography view classification. In Medical Imaging: Image-Guided Procedures, Robotic Interventions, and Modeling, 2019.
[42] B. S. Wessler, Z. Huang, G. M. Long, S. Pacifici, N. Prashar, S. Karmiy, R. A. Sandler, J. Z. Sokol, D. B. Sokol, M. M. Dehn, L. Maslon, E. Mai, A. R. Patel, and M. C. Hughes. Automated detection of aortic stenosis using machine learning. Journal of the American Society of Echocardiography, 2023.
[43] P. F. R. Wilson, M. Gilany, A. Jamzad, F. Fooladgar, M. N. N. To, B. Wodlinger, P. Abolmaesumi, and P. Mousavi. Self-supervised learning with limited labeled data for prostate cancer detection in high-frequency ultrasound. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 2023.
[44] Z. Xiang, Q. Zhuo, C. Zhao, X. Deng, T. Zhu, T. Wang, W. Jiang, and B. Lei. Self-supervised multi-modal fusion network for multi-modal thyroid ultrasound image diagnosis. Computers in Biology and Medicine, 2022.
[45] J. Zhang, S. Gajjala, P. Agrawal, G. Tison, L. Hallock, L. Beussink, M. Lassen, E. Fan, M. Aras, C. Jordan, K. Fleischmann, M. Melisko, A. Qasim, S. Shah, R. Bajcsy, and R. Deo. Fully automated echocardiogram interpretation in clinical practice: Feasibility and diagnostic accuracy. Circulation, 2018.
[46] J. Zhang, Q. He, Y. Xiao, H. Zheng, C. Wang, and J. Luo. Ultrasound image reconstruction from plane wave radio-frequency data by self-supervised deep neural network. Medical Image Analysis, 2021.
[47] H. Zhao, Q. Zheng, C. Teng, R. Yasrab, L. Drukker, A. T. Papageorghiou, and J. A. Noble. Towards unsupervised ultrasound video clinical quality assessment with multi-modality data. In Medical Image Computing and Computer Assisted Intervention, 2022.
[48] Y. Zhu, J. Ma, Z. Zhang, Y. Zhang, S. Zhu, M. Liu, Z. Zhang, C. Wu, X. Yang, J. Cheng, D. Ni, M. Xie, W. Xue, and L. Zhang. Automatic view classification of contrast and non-contrast echocardiography. Frontiers in Cardiovascular Medicine, 2022.
[49] A. Østvik, E. Smistad, S. A. Aase, B. O. Haugen, and L. Lovstakken. Real-time standard view classification in transthoracic echocardiography using convolutional neural networks. Ultrasound in Medicine & Biology, 2019.