A BERT-Based Model for Financial Social Media Sentiment Analysis

Authors: Josiel Delgadillo, Johnson Kinyua, Charles Mutigwe

Abstract:

The purpose of sentiment analysis is to determine the sentiment strength (e.g., positive, negative, neutral) from a textual source for good decision-making. Natural Language Processing (NLP) in domains such as financial markets requires knowledge of domain ontology, and pre-trained language models, such as BERT, have made significant breakthroughs in various NLP tasks by training on large-scale un-labeled generic corpora such as Wikipedia. However, sentiment analysis is a strong domain-dependent task. The rapid growth of social media has given users a platform to share their experiences and views about products, services, and processes, including financial markets. StockTwits and Twitter are social networks that allow the public to express their sentiments in real time. Hence, leveraging the success of unsupervised pre-training and a large amount of financial text available on social media platforms could potentially benefit a wide range of financial applications. This work is focused on sentiment analysis using social media text on platforms such as StockTwits and Twitter. To meet this need, SkyBERT, a domain-specific language model pre-trained and fine-tuned on financial corpora, has been developed. The results show that SkyBERT outperforms current state-of-the-art models in financial sentiment analysis. Extensive experimental results demonstrate the effectiveness and robustness of SkyBERT.

Keywords: BERT, financial markets, Twitter, sentiment analysis.

References:

[1] Amazon. (n.d.). Amazon Comprehend: Features. Retrieved June25, 2022 from https://aws.amazon.com/comprehend/features
[2] Amazon Web Services. (n.d.). Amazon Comprehend Developer Guide. Retrieved June 25, 2022 from https://docs.aws.amazon.com/comprehend/latest/dg/comprehend-dg.pdf.how-sentiment
[3] Dogu Araci. 2019. FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. CoRR abs/1908.10063 (2019). arXiv: 1908.10063 http://arxiv.org/abs/1908.10063
[4] Doğu Tan Araci and Zulkuf Genc. July 31, 2020. FinBERT: Financial Sentiment Analysis with BERT. Prosus AI Tech Blog. Retrieved July 1, 2022 from https://medium.com/prosus-ai-tech-blog/finbert-financial-sentiment-analysis-with-bert-b277a3607101
[5] Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A Pre-trained Language Model for Scientific Text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 3615–3620. https://doi.org/10.18653/v1/D19-1371
[6] Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Issues and Perspectives from 10,000 Annotated Financial Social Media Data. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 6106–6110. https://aclanthology.org/2020.lrec-1.749
[7] Common Crawl. (n.d.). Retrieved November 30, 2021 from https://commoncrawl.org/
[8] X. Cui, D. Lam, and A. Verma.2016. Embedded Valuein Bloomberg News and Social Sentiment Data. Technical Report.
[9] Tobias Daudert. 2020. A Multi-Source Entity-Level Sentiment Corpus for the Financial Domain: The FinLin Corpus. CoRR abs/2003.04073 (2020). https://arxiv.org/abs/2003.04073
[10] Vinicio Desola, Kevin Hanna, and Pri Nonis. 2019. FinBERT: pre-trained model on SEC filings for financial natural language tasks. Technical Report. https://doi.org/10.13140/RG.2.2.19153.89442
[11] Jacob Devlin and Ming-Wei Chang. November 2, 2018. Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing. Google AI Blog. Retrieved July 1, 2022 from https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html
[12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423
[13] Eugene F. Fama. 1965. Random Walks in Stock Market Prices. Financial Analysts Journal 21, 5 (1965), 55–59. http://www.jstor.org/stable/4469865
[14] Eugene F. Fama. 1970. Efficient Capital Markets: A Review of Theory and Empirical Work. The Journal of Finance 25, 2 (1970), 383–417. http://www.jstor.org/stable/2325486
[15] FinancialWeb. (n.d.). Retrieved November 30,2021 from https://www.finweb.com/
[16] Thomas Gaillat, Manel Zarrouk, André Freitas, and Brian Davis. 2018. The SSIX Corpora: Three Gold Standard Corpora for Sentiment Analysis in English, Spanish and German Financial Microblogs. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https://aclanthology.org/L18-1423
[17] Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 328–339. https://doi.org/10.18653/v1/P18-1031
[18] Allen H. Huang, Amy Y. Zang, and Rong Zheng. 2014. Evidence on the Information Content of Text in Analyst Reports. The Accounting Review 89, 6 (06 2014), 2151–2180. https://doi.org/10.2308/accr-50833
[19] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342 (2019).
[20] C. Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule Based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media 8, 1 (May 2014), 216–225. https://ojs.aaai.org/index.php/ICWSM/article/view/14550
[21] IBM. (n. d.). IBM Cloud API Docs: Natural Language Understanding. Retrieved June 25, 2022 from https://cloud.ibm.com/apidocs/natural-language-understanding?code=python
[22] IBM. (n. d.). Watson Natural Language Understanding: Features. Retrieved June 25, 2022, from https://www.ibm.com/cloud/watson-natural-language-understanding/details
[23] Investopedia. (n.d.). Financial Terms Dictionary. Retrieved November30, 2021 from https://www.investopedia.com/financial-term-dictionary-4769738
[24] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (sep 2019). https://doi.org/10.1093/bioinformatics/btz682
[25] Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020. FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI-20), Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4513–4519. https://doi.org/10.24963/ijcai.2020/622
[26] Reuters Ltd. 2004. Reuters Corpora (RCV1, RCV2, TRC2). National Institute of Standards and Technology. https://trec.nist.gov/data/reuters/reuters.html
[27] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius,and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65, 4 (2014), 782–796. https://doi.org/10.1002/asi.23062
[28] Financial Opinion Mining and Question Answering. 2017. https://sites.google.com/view/fiqa/
[29] Thao Truong Nguyen, François Trahay, Jens Domke, Aleksandr Drozd, Emil Vatai, Jianwei Liao, Mohamed Wahib, and Balazs Gerofi. 2022. Why globally reshuffle? Revisiting data shuffling in large scale deep learning. In IPDPS 2022: 36th International Parallel & Distributed Processing Symposium. IEEE, Lyon (virtual), France. https://hal.archives-ouvertes.fr/hal-03599740
[30] The First Workshop on Financial Technology and Natural Language Processing (FinNLP) with a Shared Task for Sentence Boundary Detection in PDF Noisy Text in the Financial Domain (FinSBD). (n. d.). https://sites.google.com/nlg.csie.ntu.edu.tw/finnlp/
[31] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, New Orleans, Louisiana, 2227–2237. https://doi.org/10.18653/v1/N18-1202
[32] Purva Huilgol. August 24, 2019. Accuracy vs. F1 score. Retrieved June 30, 2022, from https://medium.com/analytics-vidhya/accuracy-vs-f1-score-6258237beca2
[33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
[34] Reddit.(n.d.). Retrieved November 30, 2021 from https://www.reddit.com/
[35] Hassan Saif, Miriam Fernández, Yulan He, and Harith Alani. 2013. Evaluation datasets for Twitter sentiment analysis: a survey and a new dataset, the STS Gold. In 1st International Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM 2013). http://oro.open.ac.uk/40660/
[36] SentiStrength. (n.d.). Retrieved June 25, 2022 from http://sentistrength.wlv.ac.uk/
[37] StockTwits, Inc. (n.d.). Retrieved November 30, 2021 from https://stocktwits.com/
[38] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. 2017. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era. In 2017 IEEE International Conference on Computer Vision (ICCV). 843–852. https://doi.org/10.1109/ICCV.2017.97
[39] Bruno Taborda, Anade Almeida, José Carlos Dias, Fernando Batista, and Ricardo Ribeiro. 2021. Stock Market Tweets Data. https://doi.org/10.21227/g8vy-5w61
[40] Paul C. Tetlock. 2007. Giving Content to Investor Sentiment: The Role of Media in the Stock Market. The Journal of Finance 62, 3 (2007), 1139–1168. https://doi.org/10.1111/j.1540-6261.2007.01232.x
[41] Paul C. Tetlock, Maytal Saar-Tsechansky, and Sofus Macskassy. 2008. More Than Words: Quantifying Language to Measure Firms’ Fundamentals. The Journal of Finance 63, 3 (2008), 1437–1467. https://doi.org/10.1111/j.1540-6261.2008.01362.x
[42] Mike Thelwall, Kevan Buckley, Georgios Paltoglou, DiCai, and Arvid Kappas. 2010. Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology 61,12(2010),2544–2558. https://doi.org/10.1002/asi.21416
[43] Twitter, Inc. (n. d.). Retrieved November 30, 2021, from https://twitter.com/
[44] Yahoo! Finance. (n. d.). Retrieved November 30, 2021, from https://finance.yahoo.com/
[45] Yi Yang, Mark Christopher Siy UY, and Allen Huang. 2020. FinBERT: A Pre-trained Language Model for Financial Communications. arXiv e-prints (June 2020). https://doi.org/10.48550/arXiv.1908.10063
[46] Yang, Y. (n. d.). FinBERT. Retrieved June 25, 2022 from https://github.com/yya518/FinBERT/
[47] David Zimbra, Ahmed Abbasi, Daniel Zeng, and Hsinchun Chen. 2018. The State-of-the-Art in Twitter Sentiment Analysis: A Review and Benchmark Evaluation. ACM Trans. Manage. Inf. Syst. 9, 2, Article 5 (Aug 2018), 29 pages. org/10.1145/3185045
[48] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Rsulan Salakhutdinov, and Quoc V. Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. 2019. arXiv:1906.08237. https://arxiv.org/abs/1906.08237
[49] Mojtaba nabipour, Pooyan Nayyeri, Hamed Jabani, Shahab S., and Amir Mosavi. Predicting Stock Market Trends Using Machine Learning and Deep Learning Algorithms Via Continuous and Binary Data; a Comparative Analysis. (2020). DOI: 10.1109/ACCESS.2020.3015966. https://ieeexplore.ieee.org/abstract/document/9165760