{"title":"Response Quality Evaluation in Heterogeneous Question Answering System: A Black-box Approach","authors":"Goh Ong Sing, C. Ardil, Wilson Wong, Shahrin Sahib","volume":41,"journal":"International Journal of Computer and Information Engineering","pagesStart":1035,"pagesEnd":1041,"ISSN":"1307-6892","URL":"https:\/\/publications.waset.org\/pdf\/13286","abstract":"
The evaluation of the question answering system is a major research area that needs much attention. Before the rise of domain-oriented question answering systems based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when question answering systems began to be more domains specific, evaluation becomes a real issue. This is especially true when understanding and reasoning is required to cater for a wider variety of questions and at the same time achieve higher quality responses The research in this paper discusses the inappropriateness of the existing measure for response quality evaluation and in a later part, the call for new standard measures and the related considerations are brought forward. As a short-term solution for evaluating response quality of heterogeneous systems, and to demonstrate the challenges in evaluating systems of different nature, this research presents a black-box approach using observation, classification scheme and a scoring mechanism to assess and rank three example systems (i.e. AnswerBus, START and NaLURI).<\/p>\r\n","references":"[1] Benamara, F., Cooperative Question Answering in Restricted Domains:\r\nthe WEBCOOP Experiment. In Proceedings of the ACL Workshop on\r\nQuestion Answering in Restricted Domains, 2004.\r\n[2] Benamara, F. & Saint-Dizier, P., Advanced Relaxation for Cooperative\r\nQuestion Answering. In New Directions in Question Answering. MIT\r\nPress, 2004.\r\n[3] Chung, H., Han, K., Rim, H., Kim, S., Lee, J., Song, Y. & Yoon, D., A\r\nPractical QA System in Restricted Domains. In Proceedings of the ACL\r\nWorkshop on Question Answering in Restricted Domains, 2004.\r\n[4] Diekema, A., Yilmazel, O. & Liddy, E., Evaluation of Restricted\r\nDomain Question-Answering Systems. In Proceedings of the ACL\r\nWorkshop on Question Answering in Restricted Domains, 2004.\r\n[5] Facemire, J., A Proposed Metric for the Evaluation of Natural Language\r\nSystems. In Proceedings of the IEEE Energy and Information\r\nTechnologies in the Southeast, 1989.\r\n[6] Guida, G. & Mauri, G., A Formal Basis for Performance Evaluation of\r\nNatural Language Understanding Systems. Computational Linguistics,\r\n10(1):15-30, 1984.\r\n[7] Hirschman, L. & Gaizauskas, R., Natural Language Question\r\nAnswering: The View from Here. Natural Language Engineering,\r\n7(4):275-300, 2001.\r\n[8] Hermjakob, U., Parsing and Question Classification for Question\r\nAnswering. In Proceedings of the ACL Workshop on Open-Domain\r\nQuestion Answering, 2001.\r\n[9] Lin, J., Sinha, V., Katz, B., Bakshi, K., Quan, D., Huynh, D. & Karger,\r\nD., What Makes a Good Answer? The Role of Context in Question\r\nAnswering. In Proceedings of the 9th International Conference on\r\nHuman-Computer Interaction, 2003.\r\n[10] Katz, B. & Lin, J., START and Beyond. In Proceedings of the 6th World\r\nMulticonference Systemics, Cybernetics and Informatics, 2002.\r\n[11] Katz, B., Annotating the World Wide Web using Natural Language. In\r\nProceedings of the 5th Conference on Computer Assisted Information\r\nSearching on the Internet, 1997.\r\n[12] Katz, B., Felshin, S. & Lin, J., The START Multimedia Information\r\nSystem: Current Technology and Future Directions. In Proceedings of\r\nthe International Workshop on Multimedia Information Systems, 2002.\r\n[13] King, M., Evaluating Natural Language Processing Systems.\r\nCommunications of the ACM, 39(1):73-79, 1996.\r\n[14] Kwok, C., Weld, D. & Etzioni, O., Scaling Question Answering to the\r\nWeb. ACM Transactions on Information Systems, 19(3):242-262, 2001.\r\n[15] Maybury, M., Toward a Question Answering Roadmap. In Proceedings\r\nof the AAAI Spring Symposium on New Directions in Question\r\nAnswering, pp. vii-xi, 2003.\r\n[16] Moldovan, D., Pasca, M., Surdeanu, M. & Harabagiu, S., Performance\r\nIssues and Error Analysis in an Open-Domain Question Answering\r\nSystem. In Proceedings of the 40th Annual Meeting of the Association\r\nfor Computational Linguistics, 2002.\r\n[17] Srivastava, A. & Rajaraman, V., A Vector Measure for the Intelligence\r\nof a Question-Answering (Q-A) System. IEEE Transactions on\r\nSystems_Man and Cybernetics, 25(5):814-823, 1995.\r\n[18] Wong, W., Practical Approach to Knowledge-based Question\r\nAnswering with Natural Language Understanding and Advanced\r\nReasoning. Thesis (MSc), Kolej Universiti Teknikal Kebangsaan\r\nMalaysia, 2004.\r\n[19] Wong, W., Sing, G. O., Mohammad-Ishak, D. & Shahrin, S., Online\r\nCyberlaw Knowledge Base Construction using Semantic Network. In\r\nProceedings of the IASTED International Conference on Applied\r\nSimulation and Modeling, 2004a.\r\n[20] Wong, W., Sing, G. O. & Mokhtar, M., Syntax Preprocessing in\r\nCyberlaw Web Knowledge Base Construction. In Proceedings of the\r\nInternational Conference on Intelligent Agents, Web Technologies and\r\nInternet Commerce, 2004b.\r\n[21] Voorhees, E., Overview of TREC 2003. In Proceedings of the 12th Text\r\nRetrieval Conference, 2003.\r\n[22] Zheng, Z., Developing a Web-based Question Answering System. In\r\nProceedings of the 11th International Conference on World Wide Web,\r\n2002a.\r\n[23] Zheng, Z., AnswerBus Question Answering System. In Proceedings of\r\nthe Conference on Human Language Technology, 2002b.\r\n[24] Zweigenbaum, P., Question Answering in Biomedicine. In Proceedings\r\nof the 10th Conference of the European Chapter of the Association for\r\nComputational Linguistics, 2003.\r\n[25] Allen, J., Natural Language Understanding. Benjamin\/Cummins\r\nPublishing, 1995.\r\n[26] Nyberg, E. & Mitamura, T., Evaluating QA Systems on Multiple\r\nDimensions. In Proceedings of the Workshop on QA Strategy and\r\nResources, 2002.","publisher":"World Academy of Science, Engineering and Technology","index":"Open Science Index 41, 2010"}