Learning an Overcomplete Dictionary using a Cauchy Mixture Model for Sparse Decay

Authors: E. S. Gower, M. O. J. Hawksford

Abstract:

An algorithm for learning an overcomplete dictionary using a Cauchy mixture model for sparse decomposition of an underdetermined mixing system is introduced. The mixture density function is derived from a ratio sample of the observed mixture signals where 1) there are at least two but not necessarily more mixture signals observed, 2) the source signals are statistically independent and 3) the sources are sparse. The basis vectors of the dictionary are learned via the optimization of the location parameters of the Cauchy mixture components, which is shown to be more accurate and robust than the conventional data mining methods usually employed for this task. Using a well known sparse decomposition algorithm, we extract three speech signals from two mixtures based on the estimated dictionary. Further tests with additive Gaussian noise are used to demonstrate the proposed algorithm-s robustness to outliers.

Keywords: expectation-maximization, Pitman estimator, sparsedecomposition

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1075030

References:

[1] M. S. Lewicki and T. J. Sejnowski, "Learning Overcomplete Representations, " Neural Computations, vol. 12, No. 2, pp. 337-365, February 2000.
[2] M. Zhong, H. Tang, H. Cheng and Y. Tang, "An EM Algorithm for Learning Sparse and Overcomplete Representations, " Neurocomputing, vol. 57, pp. 467-476, 2004.
[3] M. Aharon, M. Elad and A. Bruckstein, "K-SVD: An algorithm for Designing Overcomplete Dictionaries for Sparse Representations," IEEE Transactions on Signal Processing, vol. 54, No. 11, November 2006.
[4] K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Egan, T. Lee and T. J. Sejnowski, "Dictionary Learning Algorithms for Sparse Representations, " eural Computations, vol. 15, No. 2, pp. 349-396, 2003.
[5] K. Egan, S. O. Aase and J. H. Hakon-Husoy, "Method of Optimal Directions for Frame Design, "in IEEE International Conference of Acoustic, Speech and Signal Processing, Vol. 5, pp. 2443-2446, 1999.
[6] I. T. Jollife, Principal Component Analysis, Series: Springer Series in Statistics, 2nd Edition, .
[7] P. Common, "Independent Component Analysis: A new Concept?, " Signal Processing, vol. 36, pp. 287-314, 1994.
[8] G. Marsaglia, "Ratios of Normal Variables, " Journal of Statistical Software, vol. 16, No. 4, May 2006.
[9] W. Feller, An Introduction to Probability Theory and Applications, 2nd Edition, New York: Wiley, pp. 229-235, 1968.
[10] P. G. Hoel, Introduction to Mathematical Statistics, 3rd Edition, New York: Wiley, pp. 57-62, 1962.
[11] J. W. Harris and H. Stoker, "Maximum Likelihood Method, " in Handbook of Mathematics and Computational Science, New York: Springer-Verlag, pp. 824-835, 1998.
[12] A. Koutrouvelis, "Estimation of the Location and Scale of the Cauchy Distribution using the Empirical Characteristic Function, " Biometrica, Vol. 69, pp. 205-213, 1982.
[13] F. Nagy, "Parameter Estimation of the Cauchy Distribution in Information Theory Approach, "Journal of Universal Computer Science, Vol. 12, No. 9, pp. 1332-1344, May 2006.
[14] G. B. Freue, "The Pitman Estimator of the Cauchy Location Parameter, " Journal of Statistical Planning and Inference, vol. 137, pp. 1900-1913, May 2006.
[15] K. M. Hanson and D. R. Wolf, "Estimators for the Cauchy Distribution, " In Maximum Entropy and Bayesian Methods, pp. 255-263, 1996.
[16] C. Jutten, G. H. Mohimani and M. B. Zadeh, "Fast Sparse Representations based on the l0 Norm, " In Pro. ICA-07, London, UK, 2007.
[17] G. H. Mohimani, M. B. Zaden and C. Jutten, "Complex Valued Sparse Representations based on Smoothed l0 Norm, " In ICASSP-08, 2008.
[18] S. S. Chen, D. L. Donoho and M. A. Saunders, "Atomic Decomposition by Basis Pursuit, " SIAM Journal of Computing, Vol. 20, No. 1, pp. 33- 61, 1999.
[19] S. G. Mallat and Z. Zhang, "Matching Pursuit with Time-Frequency Dictionaries, " IEEE Transactions on Signal Processing, pp. 3397-3415, December 1993.
[20] I. F. Gorodnitsky and B. D. Rao, "Sparse Signal Reconstruction from Limited Data using FOCUSS: A Re-weighted Minimum Norm Algorithm, " IEEE Transactions on Signal Processing, Vol. 45, No. 3, March 1997.
[21] J. E. Marsden and J. M. Hoffman, Basic Complex Analysis, 3rd Edition, W. H. Freeman, 1998, ISBN: 978-0716728771.