Modular Workflow System for HPC Applications

Authors: Y. Yudin, T. Krasikova, Y. Dorozhko, N. Currle-Linde

Abstract:

Nowadays, HPC, Grid and Cloud systems are evolving very rapidly. However, the development of infrastructure solutions related to HPC is lagging behind. While the existing infrastructure is sufficient for simple cases, many computational problems have more complex requirements.Such computational experiments use different resources simultaneously to start a large number of computational jobs.These resources are heterogeneous. They have different purposes, architectures, performance and used software.Users need a convenient tool that allows to describe and to run complex computational experiments under conditions of HPC environment. This paper introduces a modularworkflow system called SEGL which makes it possible to run complex computational experiments under conditions of a real HPC organization. The system can be used in a great number of organizations, which provide HPC power. Significant requirements to this system are high efficiency and interoperability with the existing HPC infrastructure of the organization without any changes.

Keywords: HPC, Molecular Dynamics, Workflow Languages, Workflow Management.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1057107

References:

[1] TOP 500, http://www.top500.org
[2] C. Byun, C. Duncan, andS. Burks, "A Comparison of Job Management Systems in Supporting HPC ClusterTools", Presentation for SUPerG, Vancouver, Canada, 2000.
[3] OpenSSH, http://www.openssh.com/
[4] I. Taylor, E. Deelman, D. Gannon, andM. Shields, Workflows for e- Science. Springer Press, 2007.
[5] N. Currle-Linde, F. Boes, P. Adamidis, and M. Resch, "GriCoL: A Language for Scientific Grids", In Proceedings of the 2nd IEEE International Conference on e-Science and Grid Computing (ESCIENCE '06), Amsterdam, Netherlands, 2006.
[6] M. Sonntag, N. Currle-Linde, K. Goerlach, and D. Karastoyanova, "Towards Simulation Workflows With BPEL: Deriving Missing Features from GriCoL", In Proceedings of the 21st IASTED International Conference on Modelling and Simulation,Banff, Alberta, Canada, 2010.
[7] I. Foster, I. Kesselman, andS. Tuecke, "The Anatomy of the Grid: Enabling Scalable Virtual Organizations",Int. J. High Perform. Comput. Appl. 15 3, 2001.
[8] R. Baker, D. Yu, andT. Wlodek, A Model for Grid User Management, Computing in High Energy and Nuclear Physics. La Jolla, California, USA, 2003.
[9] T. Antoni, W. B├╝hler, H. Dres, G. Grein, andM. Roth, "Global grid user support - building a worldwide distributed user support infrastructure", Journal of Physics: Conference Series, 2008.
[10] E. Deelman, D. Gannon, M. Shields, and I. Taylor, "Workflows and e- Science: An overview of workflow system features and capabilities",Future Generation Computer Systems, Volume: 25, Issue: 5 (2009), 528-540.
[11] C. Goble and D. De Roure, "The impact of workflow tools on datacentric research", In Data Intensive Computing: The Fourth Paradigm of Scientific Discovery, T. Hey, S. Tansley, and K. Tolle, Ed. Microsoft Research, 137-145, 2009
[12] S. Callaghan, E. Deelman, D. Guntere, G. Juve, P. Maechling, C. Brooks, K. Vahi, K. Milner, R. Gravesc, E. Field, D. Okaya, and T. Jordan,"Scaling up workflow-based applications",Journal of Computer and System Sciences, 428-446, 2010
[13] V. S. Kumar, P. Sadayappan, G. Mehta, K. Vahi, E. Deelman, V. Ratnakar, J. Kim, Y. Gil, M. W. Hall, T. M. Kurc, and J. H. Saltz,"An integrated framework for performance-based optimization of scientific workflows", In Proceedings of the 18th ACM international symposium on High performance distributed computing (HPDC '09) (Munich, Germany, June 11-13, 2009), 177-186.
[14] T. Fahringer, R. Prodan, R. Duan, F. Nerieri, S. Podlipnig, J. Qin, M. Siddiqui, H.-L. Truong, A. Villazon, and M. Wieczorek, "ASKALON: A Grid Application Development and Computing Environment", In 6th International Workshop on Grid Computing (New York 2005), 122-131.
[15] M. Wieczorek, R. Prodan, and T. Fahringer,Scheduling of Scientific Workflows in the ASKALON Grid Environment.ACM SIGMOD Record, 2005.
[16] Y. Dorozhko, T. Krasikova, Y. Yudin, N. Currle-Linde, andM. Resch, "An Abstract Language and Environment for the Creation and Execution of Experiments over Distributed Computers", InProceedings. of the International Scientific Conference Simulation-2010, Kiev, Ukraine, 2010.
[17] H. Bouziane, N. Currle-Linde, C. Perez, andM. Resch, "Analysis f Component Model Extensions to support the GriCoL Language", In Making grids Work, pp 45-59, Springer, 2008.
[18] GLUE Specification v. 2.0, http://www.ogf.org/documents/GFD.147.pdf
[19] Job Submission Description Language (JSDL) Specification, Version 1.0, http://www.gridforum.org/documents/GFD.56.pdf
[20] B. Armstrong, H. Bae, R. Eigenmann, F. Saied, M. Sayeed, andY. Zheng, "HPC Benchmarking and Performance valuation with Realistic Applications", 2006 SPEC Benchmark Workshop (spec), 2006.
[21] Java WebStart Overview, http://www.oracle.com/technetwork/java/ javase/overview-137531.html
[22] Lustre Cluster FS, http://www.lustre.org/
[23] sudo command, http://www.gratisoft.us/sudo/sudo.man.html