Data Flow Prioritization for Scientific Workflows Using A Virtual SDX on ExoGENI
Anirban Mandal, Paul Ruth, Ilya Baldin, Rafael Ferreira da Silva, Ewa Deelman
Abstract. We presented a novel, dynamically adaptable networked cloud infrastructure driven by the demand of a data-driven scientific workflow running on dynamically provisioned ‘slices’ spanning multiple ExoGENI racks that are interconnected using dynamically provisioned connections from Internet2 and ESnet. We showed how a virtual Software Defined Exchange (SDX) platform, instantiated on ExoGENI, provides additional functionality for management of scientific workflows. We demonstrated how tools developed in the DoE Panorama project can enable the Pegasus Workflow Management System to monitor and manipulate network connectivity and performance. We used a representative, data-intensive genome science workflow as a driving use case to showcase the above capabilities.