On Musical Information Geometry with Applications to Sonified Image Analysis

Authors: Shannon Steinmetz, Ellen Gethner

Abstract:

In this paper a theoretical foundation is developed to segment, analyze and associate patterns within audio. We explore this on imagery via sonified audio applied to our segmentation framework. The approach involves a geodesic estimator within the statistical manifold, parameterized by musical centricity. We demonstrate viability by processing a database of random imagery to produce statistically significant clusters of similar imagery content.

Keywords: Sonification, musical information geometry, image content extraction, automated quantification, audio segmentation, pattern recognition.

References:

[1] Nassar H Abdel-All and EI Abdel-Galil. Numerical treatment of geodesic differential. In International Mathematical Forum, volume 8, pages 15–29, 2013.
[2] Md Akhtaruzzaman and Amir A Shafie. Geometrical substantiation of phi, the golden ratio and the baroque of nature, architecture, design and engineering. International Journal of Arts, 1(1):1–22, 2011.
[3] S-I Amari. Information geometry on hierarchy of probability distributions. IEEE transactions on information theory, 47(5):1701–1711, 2001.
[4] Shun-ichi Amari. Differential geometry of a parametric family of invertible linear systems-riemannian metric, dual affine connections, and divergence. Mathematical systems theory, 20(1):53–82, 1987.
[5] Shun-Ichi Amari. Information geometry and its applications: Convex function and dually flat manifold. In LIX Fall Colloquium on Emerging Trends in Visual Computing, pages 75–102. Springer, 2008.
[6] Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American Mathematical Soc., 2007.
[7] Ariel Caticha. The basics of information geometry. In AIP Conference Proceedings, volume 1641, pages 15–26. American Institute of Physics, 2015.
[8] William WS Chen and Samuel Kotz. The riemannian structure of the three-parameter gamma distribution. 2013.
[9] Arshia Cont, Shlomo Dubnov, and G´erard Assayag. On the information geometry of audio streams with applications to similarity computing. IEEE Transactions on Audio, Speech, and Language Processing, 19(4):837–846, 2010.
[10] Arnaud Dessein and Arshia Cont. An information-geometric approach to real-time audio segmentation. IEEE Signal Processing Letters, 20(4):331–334, 2013.
[11] Jonathan T Foote and Matthew L Cooper. Media segmentation using self-similarity decomposition. In Storage and Retrieval for Media Databases 2003, volume 5021, pages 167–175. International Society for Optics and Photonics, 2003.
[12] Siddharth Gururani, Cameron Summers, and Alexander Lerch. Instrument activity detection in polyphonic music using deep neural networks. In ISMIR, pages 569–576, 2018.
[13] Thomas Hermann, Andy Hunt, and John G Neuhoff. The sonification handbook. Logos Verlag Berlin, 2011.
[14] Joachim Kopp, Ranjan Laha, Toby Opferkuch, and William Shepherd. Cuckoo’s eggs in neutron stars: can ligo hear chirps from the dark sector? Journal of High Energy Physics, 2018(11):96, 2018.
[15] Luis Fernando Abanto Le´on, Guillermo Kemper V´asquez, and Joel Telles. A novel fuzzy logic-based metric for audio quality assessment: Objective audio quality assessment. In CONATEL 2011, pages 1–10. IEEE, 2011.
[16] Tao Li and Mitsunori Ogihara. Toward intelligent music information retrieval. IEEE Transactions on Multimedia, 8(3):564–574, 2006.
[17] Frank Nielsen. An elementary introduction to information geometry. arXiv preprint arXiv:1808.08271, 2018.
[18] Konstantina Orlandatou. Sound characteristics which affect attributes of the synaesthetic visual experience. Musicae Scientiae, Vol. 19(4), pages 389–401, 2015.
[19] Renato Panda, Ricardo Manuel Malheiro, and Rui Pedro Paiva. Audio features for music emotion recognition: a survey. IEEE Transactions on Affective Computing, 2020.
[20] Tae Hong Park and Sumanth Srinivasan. The sound analysis toolbox (SATB). Ann Arbor, MI: Michigan Publishing, University of Michigan Library, 2016.
[21] CR Rao. Information and accuracy attainable in the estimation of statistical parameters. kotz s & johnson nl (eds.), breakthroughs in statistics volume i: Foundations and basic theory, 235–248, 1945.
[22] Markus Schedl, Emilia G´omez Guti´errez, and Juli´an Urbano. Music information retrieval: Recent developments and applications. Foundations and Trends in Information Retrieval. 2014 Sept 12; 8 (2-3): 127-261., 2014.
[23] Chetan Pratap Singh and T Kishore Kumar. Efficient pitch detection algorithms for pitched musical instrument sounds: A comparative performance evaluation. In 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 1876–1880. IEEE, 2014.
[24] Shannon Steinmetz. Sonic imagery: A view of music via mathematical computer science and signal processing, pages 12–15. University of Colorado at Denver, 2016.
[25] Shannon Steinmetz. Sonic imagery: A view of music via mathematical computer science and signal processing. University of Colorado at Denver, 2016.
[26] Ke Sun and St´ephane Marchand-Maillet. An information geometry of statistical manifold learning. In International Conference on Machine Learning, pages 1–9, 2014.
[27] Dmitri Tymoczko. A geometry of music. Oxford University Press, 1 edition, 2011.
[28] Robert van Gend. The fibonacci sequence and the golden ratio in music. Notes on Number Theory and Discrete Mathematics, 20(1):72–77, 2014.
[29] Robert S Wolff. Sounding out images. Computers in Physics, 6(3):287–289, 1992.