Cylc, a Python based workflow engine and meta-scheduler

The workflow engine Cylc (“silk”) manages a set of dependent tasks that need to run in a given order and deal with exceptions. It specialises in continuous workflows of cycling tasks such as those used in weather and climate forecasting and research (i.e. tasks can repeat at particular time intervals and can be triggered off the wall clock time as well as other tasks). Cylc is also easy to use with non-cycling systems. Cylc was created by Hilary Oliver at NIWA. Its core team now includes Hilary as well as members in the Modelling Infrastructure Support Systems Team at the Met Office.

Cylc was developed as a generic tool to help with the increased complexity of workflows used in the weather and climate communities and to replace the increasingly complex script-based solutions typically used within the community. It is currently being used for a very wide range of requirements from research to real-time operations including ensemble prediction systems.  Cylc is also used for a wide variety of workflows such as generation of input data, assimilation of observational data, modelling post processing and commercial product generation.