Containertask

[jgolob]

I have modified sciluigi to allow ContainerTasks (ie tasks that run in a container, using docker, singularity on HPC clusters, or AWS), including some file handling work to make this all glue together (with a new ContainerTargetInfo class).

This uses an ini-style config file in ~/.sciluigi/containerinfo.ini to allow a given system to specify the container engine / relevant settings (allocation / queue / AWS credentials) to run workflows.

This remains backwards compatible with non containerized tasks (and a workflow can mix / match them).

This version of SL also works with the current version of luigi.

An example workflow / package that uses this fork can be found at https://github.com/jgolob/maliampi

[samuell]

Hi @jgolob , this looks like massive work, thank's a lot for willing to contribute!

We will need to get some continuous integration set up to be able to safely merge things. This has been on hold too long already, but we are looking into doing it within a month or so.

Regardging the PR, I feel it is a bit too much in one PR for me to be comfortable with merging it all. Could you perhaps try to split it up into multiple PRs, based on feature or main focus area or so?

[jgolob]

Regardging the PR, I feel it is a bit too much in one PR for me to be comfortable with merging it all. Could you perhaps try to split it up into multiple PRs, based on feature or main focus area or so?

Thank you for considering the PR!

There really is one major feature added here: (Containerized tasks / targets). Each depends intimately upon the other, so I cannot split them into distinct PRs. I defer to you on your project, but it isn't clear to me how to accommodate the request to split into distinct PRs.