Parsl
Quick for “Parallel Scripting Library,” Parsl enables you to take computing jobs and break up them throughout a number of methods utilizing roughly the identical syntax as Python’s current Pool objects. It additionally enables you to sew collectively completely different computing duties into multi-step workflows, which might run in parallel, in sequence, or through map/cut back operations.
Parsl enables you to execute native Python functions, but additionally run every other exterior utility by the use of instructions to the shell. Your Python code is written like regular Python code, save for a particular perform decorator that marks the entry level to your work. The job-submission system additionally offers you fine-grained management over how issues run on the targets—for instance, the variety of cores per employee, how a lot reminiscence per employee, CPU affinity controls, how typically to ballot for timeouts, and so forth.
One wonderful characteristic Parsl gives is a set of prebuilt templates to dispatch work to a wide range of high-end computing assets. This not solely consists of staples like AWS or Kubernetes clusters, however supercomputing assets (assuming you could have entry) like Blue Waters, ASPIRE 1, Frontera, and so forth. (Parsl was co-developed with the help of most of the establishments that constructed such {hardware}.)
