Commenced in January 2007
Frequency: Monthly
Edition: International
Paper Count: 31097
An Adaptive Hand-Talking System for the Hearing Impaired

Authors: Zhou Yu, Jiang Feng


An adaptive Chinese hand-talking system is presented in this paper. By analyzing the 3 data collecting strategies for new users, the adaptation framework including supervised and unsupervised adaptation methods is proposed. For supervised adaptation, affinity propagation (AP) is used to extract exemplar subsets, and enhanced maximum a posteriori / vector field smoothing (eMAP/VFS) is proposed to pool the adaptation data among different models. For unsupervised adaptation, polynomial segment models (PSMs) are used to help hidden Markov models (HMMs) to accurately label the unlabeled data, then the "labeled" data together with signerindependent models are inputted to MAP algorithm to generate signer-adapted models. Experimental results show that the proposed framework can execute both supervised adaptation with small amount of labeled data and unsupervised adaptation with large amount of unlabeled data to tailor the original models, and both achieve improvements on the performance of recognition rate.

Keywords: sign language recognition, signer adaptation, eMAP/VFS, polynomial segment model

Digital Object Identifier (DOI):

Procedia APA BibTeX Chicago EndNote Harvard JSON MLA RIS XML ISO 690 PDF Downloads 1475


[1] I. C. Yoo and D. Yook, "Automatic sound recognition for the hearing impaired," IEEE Trans. Consumer Electron., vol. 54, no. 4, pp. 2029- 2036, Nov. 2008.
[2] 25053452.htm (in Chinese)
[3] W. Gao, Y. Chen, G. Fang, C. Yang, D. Jiang, and et al., "HandTalker II: a Chinese sign language recognition and synthesis system," in Proc. The 8th Int. Conf. on Control, Automation, Robotics and Vision (ICARCV 2004), pp. 759-764, 2004.
[4] L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proceedings of the IEEE, vol. 77, no.2, pp. 257-286, 1989.
[5] J. L. Gauvain and C. H. Lee, "Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains," IEEE Trans. Speech Audio Process., vol. 2, no. 2, pp. 291-298, Apr. 1994.
[6] B. J. Frey and D. Dueck, "Clustering by passing messages between data points," Science, vol. 315, no. 5814, pp. 972-976, 2007.
[7] J. Takahashi and S. Sagayama, "Vector-field-smoothed Bayesian learning for incremental speaker adaptation," in International Conference on Acoustics, Speech, and Signal Processing, pp. 696-699, 1995.
[8] C. F. Li, M. H. Siu, and J. S. K. Au-Yeung, "Recursive likelihood evaluation and fast search algorithm for polynomial segment model with application to speech recognition," IEEE Trans. Audio Speech Lang. Process., vol. 14, no.5, pp. 1704-1718, 2006.
[9] C. Wang, X. Chen, and W. Gao, "Generating data for signer adaptation," in Proc. Gesture Workshop, pp. 114-121, 2007.
[10] X. Zhu, "Semi-supervised learning literature survey," Computer Sciences Technical Reports 1530, University of Wisconsin Madison, 2008.