Contextual SenSe Model: Word Sense Disambiguation Using Sense and Sense Value of Context Surrounding the Target

Authors: Vishal Raj, Noorhan Abbas

Abstract:

Ambiguity in NLP (Natural Language Processing) refers to the ability of a word, phrase, sentence, or text to have multiple meanings. This results in various kinds of ambiguities such as lexical, syntactic, semantic, anaphoric and referential. This study is focused mainly on solving the issue of Lexical ambiguity. Word Sense Disambiguation (WSD) is an NLP technique that aims to resolve lexical ambiguity by determining the correct meaning of a word within a given context. Most WSD solutions rely on words for training and testing, but we have used lemma and Part of Speech (POS) tokens of words for training and testing. Lemma adds generality and POS adds properties of word into token. We have designed a method to create an affinity matrix to calculate the affinity between any pair of lemma_POS (a token where lemma and POS of word are joined by underscore) of given training set. Additionally, we have devised an algorithm to create the sense clusters of tokens using affinity matrix under hierarchy of POS of lemma. Furthermore, three different mechanisms to predict the sense of target word using the affinity/similarity value are devised. Each contextual token contributes to the sense of target word with some value and whichever sense gets higher value becomes the sense of target word. So, contextual tokens play a key role in creating sense clusters and predicting the sense of target word, hence, the model is named Contextual SenSe Model (CSM). CSM exhibits a noteworthy simplicity and explication lucidity in contrast to contemporary deep learning models characterized by intricacy, time-intensive processes, and challenging explication. CSM is trained on SemCor training data and evaluated on SemEval test dataset. The results indicate that despite the naivety of the method, it achieves promising results when compared to the Most Frequent Sense (MFS) model.

Keywords: Word Sense Disambiguation, WSD, Contextual SenSe Model, Most Frequent Sense, part of speech, POS, Natural Language Processing, NLP, OOV, out of vocabulary, ELMo, Embeddings from Language Model, BERT, Bidirectional Encoder Representations from Transformers, Word2Vec, lemma_POS, Algorithm.

References:

[1] Zhi Zhong and Hwee Tou Ng. 2012. Word Sense Disambiguation Improves Information Retrieval. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Vol. 1, pages 273–282.
[2] Yee Seng Chan, Hwee Tou Ng, and David Chiang. 2007. Word Sense Disambiguation Improves Statistical Machine Translation. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 33–40.
[3] Hung, C., & Chen, S.-J. 2016. Word sense disambiguation-based sentiment lexicons for sentiment classification. Knowledge-Based Systems, Vol. 110, pages 224-232.
[4] Hung, Jason & Wang, Ching-Sheng & Yang, Che-Yu & Chiu, Mao-Shuen & Yee, George. 2005. Applying Word Sense Disambiguation to Question Answering System for e-Learning, 19th International Conference on Advanced Information Networking and Applications, Vol. 1, pages 157 – 162.
[5] Rahman, N., Borah, B. 2020. Improvement of query-based text summarization using word sense disambiguation. Complex & Intelligent System, Vol. 6, pages 75–85.
[6] Lesk, M. 1986. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. SIGDOC '86: Proceedings of the 5th annual international conference on Systems documentation, pages 24-26.
[7] Eneko Agirre and Aitor Soroa. 2009. Personalizing PageRank for Word Sense Disambiguation. Proceedings of the 12th Conference of the European Chapter of the ACL, pages 33–41.
[8] George A. Miller. 1995. WordNet: a lexical database for English. Communication of the ACM, Vol. 38, pages 39–41.
[9] Satanjeev Banerjee and Ted Pedersen. 2002. An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet, Lecture Notes in Computer Science, Vol. 2276, Pages 136–145.
[10] Silberer, C. and Ponzetto, S. P. 2010. UHD: Cross-Lingual Word Sense Disambiguation Using Multilingual Co-occurrence Graphs. Proceedings of the 5th International Workshop on Semantic Evaluation, ACL, pages 134–137.
[11] Hwee Tou Ng and Hian Beng Lee. 1996. Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach. Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 40- 47.
[12] Hinrich Schütze. 1998. Automatic Word Sense Discrimination. Computational Linguistics, Vol. 24, No. 1, pages 97–123.
[13] David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics, pages 189–196.
[14] Tanja Gaustad. 2004. A Lemma-Based Approach to a Maximum Entropy Word Sense Disambiguation System for Dutch. Proceedings of the 20th International Conference on Computational Linguistics, pages 778–784.
[15] Cheng Niu, Wei Li, Rohini K. Srihari, Huifeng Li, and Laurie Crist. 2004. Context clustering for Word Sense Disambiguation based on modeling pairwise context similarities. Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text (ACL), pages 187–190.
[16] Nameh, M. S., Fakhrahmad, M., Jahromi, M.Z. 2011. A New Approach to Word Sense Disambiguation Based on Context Similarity. Proceedings of the World Congress on Engineering, Vol. 1, pages 456-459.
[17] Mikolov, Tomas & Chen, Kai & Corrado, G.s & Dean, Jeffrey. 2013. Efficient Estimation of Word Representations in Vector Space. Proceedings of Workshop at ICLR. 2013, arXiv preprint arXiv:1301.3781.
[18] Pennington, J., Socher, R., Manning, C.D., 2014. ‘‘Glove: Global vectors for word representation,” in Empirical Methods in Natural Language Processing, pages 1532–1543.
[19] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, pages 2227–2237.
[20] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, pages 4171–4186.
[21] Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Vol. 1, pages 99–110.
[22] Rami Al-Rfou’, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed Word Representations for Multilingual NLP. Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192.
[23] Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750.
[24] George A Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G Thomas. 1994. Using a semantic concordance for sense identification. Proceedings of the workshop on Human Language Technology, pages 240–243.
[25] Eneko Agirre, Oier Lopez de Lacalle, Christiane Fellbaum, Shu-Kai Hsieh, Maurizio Tesconi, Monica Monachini, Piek Vossen, and Roxanne Segers. 2010. SemEval-2010 Task 17: All-Words Word Sense Disambiguation on a Specific Domain. Proceedings of the 5th International Workshop on Semantic Evaluation, pages 75–80
[26] Zhi Zhong and Hwee Tou Ng. 2010. It Makes Sense: A Wide-Coverage Word Sense Disambiguation System for Free Text. Proceedings of the ACL 2010 System Demonstrations, pages 78–83.
[27] Philip Edmonds and Scott Cotton. 2001. Senseval-2: Overview. Proceedings of The Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1–6.
[28] Benjamin Snyder and Martha Palmer. 2004. The English all-words task. Proceedings of the 3rd International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 41–43.
[29] Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task-17: English lexical sample, SRL and all words. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 87–92.
[30] Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 Task 12: Multilingual Word Sense Disambiguation. Proceedings of SemEval 2013, pages 222–231.
[31] Andrea Moro and Roberto Navigli. 2015. SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288-297.
[32] Cucerzan, R.S., C. Schafer, and D. Yarowsky. 2002. Combining classifiers for word sense disambiguation. Natural Language Engineering, Vol. 8, No. 4, pages 327- 341.