Commenced in January 2007
Paper Count: 31108
Design and Implementation of Shared Memory based Parallel File System Logging Method for High Performance Computing
Abstract:I/O workload is a critical and important factor to analyze I/O pattern and file system performance. However tracing I/O operations on the fly distributed parallel file system is non-trivial due to collection overhead and a large volume of data. In this paper, we design and implement a parallel file system logging method for high performance computing using shared memory-based multi-layer scheme. It minimizes the overhead with reduced logging operation response time and provides efficient post-processing scheme through shared memory. Separated logging server can collect sequential logs from multiple clients in a cluster through packet communication. Implementation and evaluation result shows low overhead and high scalability of this architecture for high performance parallel logging analysis.
Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1082649Procedia APA BibTeX Chicago EndNote Harvard JSON MLA RIS XML ISO 690 PDF Downloads 1283
 John K. Ousterhout, Hervg Da Costa, David Harrison, John A. Kunze, Mike Kupfer, and James G. Thompson, " A Trace-Driven Analysis of the UNIX 4.2 BSD File System," ACM SIGOPS Operating Systems Review archive, Volume 19, Issue 5, pp. 15~24, 1985.
 Drew Roselli, Jacob R. Lorch, and Thomas E. Anderson, "A comparison of file system workloads," Proc. of USENIX Annual Technical Conference, pp. 41~54, 2000.
 Akshat Aranya, Charles P. Wright, and Erez Zadok, "Tracefs: A File System to Trace Them All," FAST 2004.
 Pin Lu and Kai Shen, "Multi-Layer Event Trace Analysis for Parallel I/O Performance Tuning," Proceedings of the 2007 International Conference on Parallel Processing, 2007.
 Anthony Chan, William Gropp, and Ewing Lusk, "User-s Guide for MPE: Extensions for MPI Programs", from MPICH2 web site, http://www.mcs.anl.gov/research/projects/mpich2
 MPICH2 web site, http://www.mcs.anl.gov /research/projects/mpich2
 Rajeev Thakur, William Gropp, Ewing Lusk, "On Implementing MPI-IO Portably and with High Performance," In Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems, pp. 23-32, 1999.
 PVFS web size, http://www.pvfs.org
 Lustre web site, http://wiki.lustre.org
 GPFS Wikipedia, http://en.wikipedia.org/wiki/GPFS
 Chris Ruemmler and John Wilkes, "A trace-driven analysis of disk working set sizes", Technical Report HPL-OSR-93-23, Hewlett- Packard Laboratories, April 1993.
 sourceforge starce home page, http://sourceforge.net/projects/strace/
 Akshat Aranya, Charles P. Wright, Erez Zadok, "Tracefs: A File System to Trace Them All," Proceedings of the 3rd USENIX Conference on File and Storage Technologies, pp. 129 - 145, 2004.
 L. Mummert, M. Satyanarayanan, "Long term distributed file reference tracing: Implementation and experience," SoftwareÔÇöPractice & Experience, Volume 26, Issue 6, pp. 705-736, 1996.
 tcpdump/libcap homepage, http://www.tcpdump.org/
 M. Blaze, "NFS Tracing by Passive Network Monitoring, " In Proceedings of the USENIX Winter Conference, January 1992.
 D. Ellard, J. Ledlie, P. Malkani, and M. Seltzer. Passive NFS Tracing of Email and Research Workloads. In Proceedings of the Annual USENIX Conference on File and Storage Technologies, March 2003.
 Andrew W. Leung, Shankar Pasupathy, Garth Goodson, Ethan L. Miller "Measurement and Analysis of Large-Scale Network File System Workloads," In the proceedings of the 2008 USENIX Annual Technical Conference, June 2008.
 Ibrahim F. Haddad inSysAdmin, "PVFS: A Parallel Virtual File System for Linux Cluster," Linux Journal, 2000.
 Mohan Rajagopalan, Matti Hiltunen, Trevor Jim, and Richard Schlichting, "Authenticated System Calls," DSN-2005: The International Conference on `Dependable Systems and Networks, June 2005.
 Intel web site, http://software.intel.com/en-us/forums/watercoolercatchall /topic/54276
 iozone web site, http://www.iozone.org/