COVID_ICU_BERT: A Fine-tuned Language Model for COVID-19 Intensive Care Unit Clinical Notes

Authors: Shahad Nagoor, Lucy Hederman, Kevin Koidl, Annalina Caputo

Abstract:

Doctors’ notes reflect their impressions, attitudes, clinical sense, and opinions about patients’ conditions and progress, and other information that is essential for doctors’ daily clinical decisions. Despite their value, clinical notes are insufficiently researched within the language processing community. Automatically extracting information from unstructured text data is known to be a difficult task as opposed to dealing with structured information such as physiological vital signs, images and laboratory results. The aim of this research is to investigate how Natural Language Processing (NLP) techniques and machine learning techniques applied to clinician notes can assist in doctors’ decision making in Intensive Care Unit (ICU) for coronavirus disease 2019 (COVID-19) patients. The hypothesis is that clinical outcomes like survival or mortality can be useful to influence the judgement of clinical sentiment in ICU clinical notes. This paper presents two contributions: first, we introduce COVID_ICU_BERT, a fine-tuned version of a clinical transformer model that can reliably predict clinical sentiment for notes of COVID patients in ICU. We train the model on clinical notes for COVID-19 patients, ones not previously seen by Bio_ClinicalBERT or Bio_Discharge_Summary_BERT. The model which was based on Bio_ClinicalBERT achieves higher predictive accuracy than the one based on Bio_Discharge_Summary_BERT (Acc 93.33%, AUC 0.98, and Precision 0.96). Second, we perform data augmentation using clinical contextual word embedding that is based on a pre-trained clinical model to balance the samples in each class in the data (survived vs. deceased patients). Data augmentation improves the accuracy of prediction slightly (Acc 96.67%, AUC 0.98, and Precision 0.92).

Keywords: BERT fine-tuning, clinical sentiment, COVID-19, data augmentation.

References:

[1] I. P. Lynch, P. E. Roberts, J. R. Keebler, O. Guttman, and P. E. Greilich, “Error Detection and Reporting in the Intensive Care Unit: Progress, Barriers, and Future Direction,” Current Anesthesiology Reports, vol. 7, no. 3, pp. 310–319, Sep. 2017. (Online). Available: https://doi.org/10.1007/s40140-017-0228-3
[2] W. G. Johnson, T. A. Brennan, J. P. Newhouse, L. L. Leape, A. G. Lawthers, H. H. Hiatt, and P. C. Weiler, “The economic consequences of medical injuries. Implications for a no-fault insurance plan,” JAMA, vol. 267, no. 18, pp. 2487–2492, May 1992.
[3] E. J. Thomas, D. M. Studdert, J. P. Newhouse, B. I. Zbar, K. M. Howard, E. J. Williams, and T. A. Brennan, “Costs of medical injuries in Utah and Colorado,” Inquiry: A Journal of Medical Care Organization, Provision and Financing, vol. 36, no. 3, pp. 255–264, 1999.
[4] L. L. Leape, T. A. Brennan, N. Laird, A. G. Lawthers, A. R. Localio, B. A. Barnes, L. Hebert, J. P. Newhouse, P. C. Weiler, and H. Hiatt, “The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II,” The New England Journal of Medicine, vol. 324, no. 6, pp. 377–384, Feb. 1991.
[5] H.-J. Kong, “Managing Unstructured Big Data in Healthcare System,” Healthcare Informatics Research, vol. 25, no. 1, pp. 1–2, Jan. 2019, publisher: Korean Society of Medical Informatics. (Online). Available: http://e-hir.org/journal/view.php?id=10.4258/hir.2019.25.1.1
[6] K. Huang, A. Singh, S. Chen, E. Moseley, C.-Y. Deng, N. George, and C. Lindvall, “Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation,” in Proceedings of the 3rd Clinical Natural Language Processing Workshop. Online: Association for Computational Linguistics, 2020, pp. 94–100. (Online). Available: https://www.aclweb.org/anthology/2020.clinicalnlp-1.11
[7] S. N. Kasthurirathne, B. E. Dixon, J. Gichoya, H. Xu, Y. Xia, B. Mamlin, and S. J. Grannis, “Toward better public health reporting using existing off the shelf approaches: A comparison of alternative cancer detection approaches using plaintext medical data and non-dictionary based feature selection,” Journal of Biomedical Informatics, vol. 60, pp. 145–152, Apr. 2016.
[8] S. Nuthakki, S. Neela, J. W. Gichoya, and S. Purkayastha, “Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks,” arXiv:1912.12397 (cs), Dec. 2019, arXiv: 1912.12397. (Online). Available: http://arxiv.org/abs/1912.12397
[9] Y. Wang, K. Verspoor, and T. Baldwin, “Learning from Unlabelled Data for Clinical Semantic Textual Similarity,” in Proceedings of the 3rd Clinical Natural Language Processing Workshop. Online: Association for Computational Linguistics, 2020, pp. 227–233. (Online). Available: https://www.aclweb.org/anthology/2020.clinicalnlp-1.25
[10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Association for Computational Linguistics, 2019, pp. 4171–4186.
[11] E. Alsentzer, J. Murphy, W. Boag, W.-H. Weng, D. Jindi, T. Naumann, and M. McDermott, “Publicly Available Clinical BERT Embeddings,” in Proceedings of the 2nd Clinical Natural Language Processing Workshop. Minneapolis, Minnesota, USA: Association for Computational Linguistics, Jun. 2019, pp. 72–78. (Online). Available: https://aclanthology.org/W19-1909
[12] A. E. W. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark, “MIMIC-III, a freely accessible critical care database,” Scientific Data, vol. 3, no. 1, p. 160035, May 2016, number: 1 Publisher: Nature Publishing Group. (Online). Available: https://www.nature.com/articles/sdata201635
[13] Y. Deng, T. Declerck, P. Lendvai, and K. Denecke, “The Generation of a Corpus for Clinical Sentiment Analysis,” in The Semantic Web, ser. Lecture Notes in Computer Science, H. Sack, G. Rizzo, N. Steinmetz, D. Mladeni´c, S. Auer, and C. Lange, Eds. Cham: Springer International Publishing, 2016, pp. 311–324.
[14] K. Denecke and Y. Deng, “Sentiment analysis in medical settings: New opportunities and challenges,” Artificial Intelligence in Medicine, vol. 64, no. 1, pp. 17–27, May 2015. (Online). Available: https://www.sciencedirect.com/science/article/pii/S0933365715000299
[15] I. E. R. Waudby-Smith, N. Tran, J. A. Dubin, and J. Lee, “Sentiment in nursing notes as an indicator of out-of-hospital mortality in intensive care patients,” PloS One, vol. 13, no. 6, p. e0198687, 2018.
[16] Y. Zou, J. Wang, Z. Lei, Y. Zhang, and W. Wang, “Sentiment Analysis for Necessary Preview of 30-Day Mortality in Sepsis Patients and the Control Strategies,” Journal of Healthcare Engineering, vol. 2021, p. 1713363, 2021.
[17] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to Fine-Tune BERT for Text Classification?” in Chinese Computational Linguistics, ser. Lecture Notes in Computer Science, M. Sun, X. Huang, H. Ji, Z. Liu, and Y. Liu, Eds. Cham: Springer International Publishing, 2019, pp. 194–206.
[18] Y. Zhou and V. Srikumar, “A Closer Look at How Fine-tuning Changes BERT,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 1046–1061. (Online). Available: https://aclanthology.org/2022.acl-long.75
[19] G. E. Weissman, L. H. Ungar, M. O. Harhay, K. R. Courtright, and S. D. Halpern, “Construct validity of six sentiment analysis methods in the text of encounter notes of patients with critical illness,” Journal of Biomedical Informatics, vol. 89, pp. 114–121, Jan. 2019.
[20] Q. Gao, D. Wang, P. Sun, X. Luan, and W. Wang, “Sentiment Analysis Based on the Nursing Notes on In-Hospital 28-Day Mortality of Sepsis Patients Utilizing the MIMIC-III Database,” Computational and Mathematical Methods in Medicine, vol. 2021, p. 3440778, 2021.
[21] M. Abbaspour Onari, S. Yousefi, M. Rabieepour, A. Alizadeh, and M. Jahangoshai Rezaee, “A medical decision support system for predicting the severity level of COVID-19,” Complex & Intelligent Systems, vol. 7, no. 4, pp. 2037–2051, Aug. 2021. (Online). Available: https://link.springer.com/10.1007/s40747-021-00312-1
[22] M. Chieregato, F. Frangiamore, M. Morassi, C. Baresi, S. Nici, C. Bassetti, C. Bnà, and M. Galelli, “A hybrid machine learning/deep learning COVID-19 severity predictive model from CT images and clinical data,” Scientific Reports, vol. 12, no. 1, p. 4329, Mar. 2022, number: 1 Publisher: Nature Publishing Group. (Online). Available: https://www.nature.com/articles/s41598-022-07890-1
[23] O. Kocadagli, A. Baygul, N. Gokmen, S. Incir, and C. Aktan, “Clinical prognosis evaluation of COVID-19 patients: An interpretable hybrid machine learning approach,” Current Research in Translational Medicine, vol. 70, no. 1, p. 103319, Jan. 2022.
[24] L. Yan, H.-T. Zhang, J. Goncalves, Y. Xiao, M. Wang, Y. Guo, C. Sun, X. Tang, L. Jing, M. Zhang, X. Huang, Y. Xiao, H. Cao, Y. Chen, T. Ren, F. Wang, Y. Xiao, S. Huang, X. Tan, N. Huang, B. Jiao, C. Cheng, Y. Zhang, A. Luo, L. Mombaerts, J. Jin, Z. Cao, S. Li, H. Xu, and Y. Yuan, “An interpretable mortality prediction model for COVID-19 patients,” Nature Machine Intelligence, vol. 2, no. 5, pp. 283–288, May 2020, number: 5 Publisher: Nature Publishing Group. (Online). Available: https://www.nature.com/articles/s42256-020-0180-7
[25] A. Vaid, S. K. Jaladanki, J. Xu, S. Teng, A. Kumar, S. Lee, S. Somani, I. Paranjpe, J. K. De Freitas, T. Wanyan, K. W. Johnson, M. Bicak, E. Klang, Y. J. Kwon, A. Costa, S. Zhao, R. Miotto, A. W. Charney, E. Böttinger, Z. A. Fayad, G. N. Nadkarni, F. Wang, and B. S. Glicksberg, “Federated Learning of Electronic Health Records to Improve Mortality Prediction in Hospitalized Patients With COVID-19: Machine Learning Approach,” JMIR Medical Informatics, vol. 9, no. 1, p. e24207, Jan. 2021. (Online). Available: http://medinform.jmir.org/2021/1/e24207/
[26] A. Karthikeyan, A. Garg, P. K. Vinod, and U. D. Priyakumar, “Machine Learning Based Clinical Decision Support System for Early COVID-19 Mortality Prediction,” Frontiers in Public Health, vol. 9, p. 626697, 2021.
[27] J. Berenguer, A. M. Borobia, P. Ryan, J. Rodríguez-Baño, J. M. Bellón, I. Jarrín, J. Carratalà, J. Pachón, A. J. Carcas, M. Yllescas, and J. R. Arribas, “Development and validation of a prediction model for 30-day mortality in hospitalised patients with COVID-19: the COVID-19 SEIMC score,” Thorax, vol. 76, no. 9, pp. 920–929, Sep. 2021, publisher: BMJ Publishing Group Ltd Section: Respiratory infection. (Online). Available: https://thorax.bmj.com/content/76/9/920
[28] P. Schwab, A. Mehrjou, S. Parbhoo, L. A. Celi, J. Hetzel, M. Hofer, B. Schölkopf, and S. Bauer, “Real-time prediction of COVID-19 related mortality using electronic health records,” Nature Communications, vol. 12, no. 1, p. 1058, Feb. 2021.
[29] A. M. U. D. Khanday, S. T. Rabani, Q. Khan, N. Rouf, and M. M. U. Din, “Machine learning based approaches for detecting COVID-19 using clinical text data,” International journal of information technology : an official journal of Bharati Vidyapeeth’s Institute of Computer Applications and Management, 2020.
[30] J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi, “Covid-19 image data collection: Prospective predictions are the future,” arXiv 2006.11988, 2020. (Online). Available: https://github.com/ieee8023/covid-chestxray-dataset
[31] J. P. Cohen, P. Morrison, and L. Dao, “Covid-19 image data collection,” arXiv 2003.11597, 2020. (Online). Available: https://github.com/ieee8023/covid-chestxray-dataset
[32] I. Loshchilov and F. Hutter, “DECOUPLED WEIGHT DECAY REGULARIZATION,” The International Conference on Learning Representations, ICLR 2019, p. 18, 2019.
[33] S. Kobayashi, “Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations,” May 2018, arXiv:1805.06201 (cs). (Online). Available: http://arxiv.org/abs/1805.06201