A Generic Middleware to Instantly Sync Intensive Writes of Heterogeneous Massive Data via Internet

Authors: Haitao Yang, Zhenjiang Ruan, Fei Xu, Lanting Xia

Abstract:

Industry data centers often need to sync data changes reliably and instantly from a large-scale of heterogeneous autonomous relational databases accessed via the not-so-reliable Internet, for which a practical generic sync middleware of low maintenance and operation costs is most wanted. To this demand, this paper presented a generic sync middleware system (GSMS), which has been developed, applied and optimized since 2006, holding the principles or advantages that it must be SyncML-compliant and transparent to data application layer logic without referring to implementation details of databases synced, does not rely on host computer operating systems deployed, and its construction is light weighted and hence of low cost. Regarding these hard commitments of developing GSMS, in this paper we stressed the significant optimization breakthrough of GSMS sync delay being well below a fraction of millisecond per record sync. A series of ultimate tests with GSMS sync performance were conducted for a persuasive example, in which the source relational database underwent a broad range of write loads (from one thousand to one million intensive writes within a few minutes). All these tests showed that the performance of GSMS is competent and smooth even under ultimate write loads.

Keywords: Heterogeneous massive data, instantly sync intensive writes, Internet generic middleware design, optimization.

References:

[1] Haitao Yang, Peng Yang, Pingjing Lu, and Zhenghua Wang, “A SyncML Middleware-Based Solution for Pervasive Relational Data Synchronization,” IFIP Int’l Conf. on Network and Parallel Computing (NPC 2008), Shanghai, China, Oct 18-20, 2008. J. Cao et al. (Eds.): Springer, LNCS 5245, pp.308–319, 2008.
[2] Haitao Yang, Fei Xu, Zhenjiang Ruan, “Universal Multi-Source Heterogeneous Large-Scale Data Synchronization System,” Patents Registry, the Hong Kong Special Administrative Region. Patent Type: Standard Patent, Patent No.: HK1246878. Date of publication of grant of patent: 23-04-2021, Date of publication of application: 14-09-2018.
[3] ANSI X3.135-1992, ISO/IEC 9075:1992, SQL-92 (SQL2)
[4] OASIS, The SyncML Initiative, technology reports, last modified on: April 29, 2003. http://xml.coverpages.org/syncML.html
[5] Haitao Yang, Practice and Research Notes in Relational Database Applications. New York: Nova Science Publishers, Inc., 2010. ISBN: 978-1-61668-850-9
[6] Jerome Saltzer, and M. Kaashoek, RES.6-004 Principles of Computer System Design: An Introduction. Chapter 7, “The Network as a System and as a System Component.” Spring 2009. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu.
[7] Andrei F. Klein, Mihai Stefanescu, Alan Saied, and Kurt Swakhoven, “An experimental comparison of ActiveMQ and Open MQ brokers in asynchronous cloud environment,” 2015 Fifth Int’l Conf. on Digital Information Processing and Communications (ICDIPC). IEEE, 2015, pp.24-30.
[8] Apache Software Foundation, ActiveMQ, http://activemq.apache.org/
[9] Ravinder Gupta, Introduction to Oracle GoldenGate (OGG). Mastering Oracle GoldenGate. Apress, 2016.