Scientific workflows have become mainstream for conducting large-scale scientific research. As a result, many workflow applications and Workflow Management Systems (WMSs) have been developed as part of the cyberinfrastructure to allow scientists to execute their applications seamlessly on a range of distributed platforms. In spite of many success stories, building large-scale workflows and orchestrating their executions efficiently (in terms of performance, reliability, and cost) remains a challenge given the complexity of the workflows themselves and the complexity of the underlying execution platforms. A fundamental necessary next step is the establishment of a solid “experimental science” approach for future workflow technology development. Such an approach is useful for scientists who need to design workflows and pick execution platforms, for WMS developers who need to compare alternate design and implementation options, and for researchers who need to develop novel decision-making algorithms to be implemented as part of WMSs. The broad objective of this work is to provide foundational software, the Workflow Simulation Workbench (WRENCH), upon which to develop the above experimental science approach. Capitalizing on recent advances in distributed application and platform simulation technology, WRENCH makes it possible to (i) quickly prototype workflow, WMS implementations, and decision-making algorithms; and (ii) evaluate/compare alternative options scalably and accurately for arbitrary, and often hypothetical, experimental scenarios. This project will define a generic and foundational software architecture, that is informed by current state-of-the-art WMS designs and planned future designs. The implementation of the components in this architecture when taken together form a generic “scientific instrument” that can be used by workflow users, developers, and researchers. This scientific instrument will be instantiated for several real-world WMSs and used for a range of real-world workflow applications. In a particular case-study, it will be used with a popular WMS (Pegasus) to revisit published results and scheduling algorithms in the area of workflow planning optimizations. The objective is to demonstrate the benefit of using an experimental science approach for WMS research. Another impact of this project is that it makes it possible to include scientific workflow content pervasively in undergraduate and graduate computer science curricula, even for students without any access to computing infrastructure, by defining meaningful pedagogic activities that only require a computer and the WRENCH software stack. This educational impact will be demonstrated in the classroom in both undergraduate and graduate courses at our institutions.
Project Website: http://wrench-project.org/
Investigators: Henri Casanova (PI), Rafael Ferreira da Silva (Co-PI)
Start Date: January 1, 2017
End Date: December 31, 2019