Containertask#62

Open

jgolob wants to merge 101 commits into

pharmbio:masterfrom

jgolob:containertask

Commits on Apr 9, 2018

Added containertasks
Jonathan Golob
committed
Added engine to containerinfo
Jonathan Golob
committed
added singularity_slurm engine basics
Jonathan Golob
committed

Commits on Apr 10, 2018

Added bucket_command_wrapper.py to tools
Jonathan Golob
committed

Commits on Apr 11, 2018

Pre pull commit
Jonathan Golob
committed
Working bucket_command_wrapper.py
Jonathan Golob
committed

Commits on Apr 12, 2018

Changed BCW to allow repeated file commands and accept command as env variable
Jonathan Golob
committed
Added ContainerTarget
Jonathan Golob
committed
Made filesytem path just path component of url
Jonathan Golob
committed
Added ContainerTaskInfo to base import
Jonathan Golob
committed
Made containertaskinfo inherit from taskinfo
Jonathan Golob
committed

Commits on Apr 13, 2018

Partially completed AWS batch engine
Jonathan Golob
committed
working on aws batch
Jonathan Golob
committed

Commits on Apr 16, 2018

Working towards complete AWS-batch engine
Jonathan Golob
committed

Commits on Apr 20, 2018

Pre work
Jonathan Golob
committed

Commits on Apr 25, 2018

Working AWS batch engine with ability to upload / download from S3 as needed
Jonathan Golob
committed
Tidied up a bit of the logging for AWS-batch engine
Jonathan Golob
committed
Allowed for piped and && / || commands in both docker and aws_batch
Jonathan Golob
committed

Commits on Apr 26, 2018

Moved custom mounts to containerinfo
Jonathan Golob
committed
Moved mounts to a containerinfo parameter, and implemented for AWS batch
Jonathan Golob
committed
Switch from set to list
Jonathan Golob
committed
From string to bool for readonly
Jonathan Golob
committed
name for volume through uuid
Jonathan Golob
committed

Commits on Apr 27, 2018

check not just registered job def, but active
Jonathan Golob
committed
String for name not class uuid
Jonathan Golob
committed
Fixed a few bugs, and moved over to the packaged bucket_container_wrapper
Jonathan Golob
committed
Fixed bug where no files are being uploaded to S3 temp
Jonathan Golob
committed

Commits on Apr 28, 2018

Docker engine working, with switch to targets rather than paths to be faancy
Jonathan Golob
committed

Commits on May 1, 2018

basic engine for docker and aws_batch done
Jonathan Golob
committed
Effort to get basics of slurm_singularity engine working
Jonathan Golob
committed
Fixed bug in containerinfo init
Jonathan Golob
committed

Commits on May 2, 2018

Working on singularity_slurm code
Jonathan Golob
committed
Generally working slurm-sciluigi version
Jonathan Golob
committed
Pre pull commit
Jonathan Golob
committed
Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask
Jonathan Golob
committed
Working singularity_slurm code
Jonathan Golob
committed
Continued refinement of slurm_singularity engine
Jonathan Golob
committed
Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask
Jonathan Golob
committed

Commits on May 3, 2018

Catch ClientError for AWS Batch API calls
Sam Minot
committed
Add MAX_BOTO_TRIES
Sam Minot
committed
Merge branch 'catch_batch_client_exceptions' of https://github.com/jgolob/sciluigi into containertask
Jonathan Golob
committed
Pre pull commit
Jonathan Golob
committed
Pre-pull commit
Jonathan Golob
committed
Switch boto_max_tries to a containerinfo variable
Jonathan Golob
committed
Merge branch 'catch_batch_client_exceptions' into containertask
Jonathan Golob
committed
Fixed a minor bug when checking batch job status fails
Jonathan Golob
committed

Commits on May 10, 2018

Add custom aws batch job name param
Sam Minot
committed
Change to "_prefix" to match functionality
Sam Minot
committed
Merge pull request #3 from jgolob/custom_aws_batch_job_name
jgolob
authored

Commits on May 11, 2018

Mild changes
Jonathan Golob
committed

Commits on May 14, 2018

Fixed bug involving need to make directories when batch -> local FS
Jonathan Golob
committed

Commits on May 17, 2018

Commits on May 25, 2018

Parameterized aws batch job poll time.
Jonathan Golob
committed
Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask
Jonathan Golob
committed
removed extra param
Jonathan Golob
committed
commonprefix -> commonpath to fix a bug
Jonathan Golob
committed
Fixed bug in pol vs poll
Jonathan Golob
committed

Commits on Jun 27, 2018

Updated the readme to document how containers work now.
Jonathan Golob
committed
Added dependencies
Jonathan Golob
committed

Commits on Aug 7, 2018

Permissions changes. setup.py modified to better fit actual dependencies

Jonathan Golob
authored and
Jonathan Golob
committed

Commits on Aug 15, 2018

Working version for singularity on PBS.
Jonathan Golob
committed

Commits on Aug 28, 2018

Many changes to optimize with PBS-singularity
Jonathan Golob
committed

Commits on Sep 5, 2018

Fixes for PBS
Jonathan Golob
committed
Merge branch 'pbs' into containertask

Jonathan Golob
authored and
Jonathan Golob
committed

Commits on Sep 6, 2018

Added example for ContainerTask as well as an example Dockerfile to make a container

Jonathan Golob
authored and
Jonathan Golob
committed

Commits on Sep 18, 2018

Working example plus new class to poll AWS

Jonathan Golob
authored and
Jonathan Golob
committed
Working with a few bits of fuss batch poller

Jonathan Golob
authored and
Jonathan Golob
committed
Working consolidation of polling for job status on AWS

Jonathan Golob
authored and
Jonathan Golob
committed
Merge branch 'polling_aws' into containertask

Jonathan Golob
authored and
Jonathan Golob
committed

Commits on Sep 28, 2018

Added ability to read containerinfo from a config file
Jonathan Golob
committed

Commits on Nov 7, 2018

Fix of singularity run commands
Jonathan Golob
committed
Merge pull request #5 from pharmbio/master
jgolob
authored
Merge pull request #6 from jgolob/master
jgolob
authored

Commits on Nov 15, 2018

Corrected loading of mounts from ini
Jonathan Golob
committed
Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask
Jonathan Golob
committed

Commits on Nov 19, 2018

Reading of config from ini more robust
Jonathan Golob
committed
Fixed bug in readconfig
Jonathan Golob
committed

Commits on Nov 20, 2018

Some debugging of aws mounts
Jonathan Golob
committed
Switch job def
Jonathan Golob
committed
Fixed loading of mounts from config
Jonathan Golob
committed

Commits on Nov 21, 2018

A bit more logging around task watcher
Jonathan Golob
committed
Moved batch_task_watcher global holder to __init__.py
Jonathan Golob
committed
Moved global
Jonathan Golob
committed
Moved batch watcher to module global
Jonathan Golob
committed

Commits on Nov 27, 2018

Change to loading of batch task watcher
Jonathan Golob
committed
Made import a bit more robust in broken AWS environments
Jonathan Golob
committed
Switch to debug for task count
Jonathan Golob
committed
back to info for polling
Jonathan Golob
committed

Commits on Nov 29, 2018

Attempt to terminate running aws jobs on workflow exit
Jonathan Golob
committed
Added lock for singularity
Jonathan Golob
committed

Commits on Feb 14, 2019

Fixed a few bugs in the slurm engine
Jonathan Golob
committed

Commits on Feb 15, 2019

Fixed a few bugs in slurm.
Jonathan Golob
committed

Commits on Feb 17, 2019

SLURM changes
Jonathan Golob
committed

Commits on Apr 19, 2019

ugly fix for interface change in luigi
Jonathan Golob
committed

Commits on Apr 23, 2019

Added sample config. Bumped version to 2.0.0
Jonathan Golob
committed

Commits on Apr 26, 2019

made easier defaults in config

Jonathan Golob
authored and
Jonathan Golob
committed

Commits on May 2, 2019

Adjustments to singularity settings.
Jonathan Golob
committed
Bumped version to 2.0.1
Jonathan Golob
committed
pre pull
Jonathan Golob
committed
Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask
Jonathan Golob
committed