v0.6.0

miniwdl run

• --dir behavior revised to be more predictable: always run in a new timestamp-named subdirectory of the specified directory, unless the latter ends with .
• Automatically download input files specified as public http[s]/ftp/gs URIs
• Honor runtime.maxRetries to retry task command or docker failures
• output_links feature expanded to apply to all nested task/workflow runs, and include Files embedded in structs and other composite output values
• Manipulate ownership/permissions to improve security and user experience:
  • as run completes, chown everything in run directory to invoking user and primary group
  • run task commands with membership in invoking user's primary group, ensuring access to working directory even if they've dropped privileges (a good docker security practice)
  • --as-me to force all task commands to run as invoking user (more secure, but blocks commands that assume root e.g. apt-get)
• Revert to always running command in bash (as required by WDL spec) rather than container $SHELL

miniwdl check

• Suppress UnnecessaryQuantifier for optional File? task outputs

v0.5.2

miniwdl run

• generates a "rerun" script in the run directory, which can be sourced from the shell to re-run with the same inputs (#149)
• use docker image's $SHELL to interpret task command, instead of hard-coding bash
• log messages have been revised and reformatted to facilitate programmatic consumption (#248)
• probable fix for a sporadic hang under erratic host performance, such as when out of memory (#263)
• based on runtime.memory, create host memory reservations to avoid overbooking parallel memory-intensive tasks (#250)
  • memory reservations affect task scheduling, but aren't enforced limits (unlike runtime.cpu, which creates both reservations and limits)
• shares thread pools to respect task concurrency limits across sub-workflow calls (#203)

Known limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)
• Scheduler ignores runtime.disks and runtime.maxTries (#251)

Please vote on these and/or report other interoperability problems via Issues!

v0.5.1

miniwdl run

• support optional File? outputs from tasks (#255, openwdl/wdl#310)
• display the available inputs/outputs in source order
• fix duplication in workflow.log file

Known limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)
• Scheduler considers each tasks's runtime.cpu (default 1) versus the available host processors, but ignores runtime.{memory,disks} (#250)
• Scheduler may exceed the -@ parallelism limit argument when sub-workflows are called, potentially up to # host processors (#203).

Please vote on these and/or report other interoperability problems via Issues!

miniwdl cromwell

• Cromwell 47

WDL

• permit unbound, optional declarations in task/workflow bodies (outside of inputs) as a way of creating a null value for later use (#253)
• WDL.Type.Any always coerces to/from optional types
• preserve Value.expr references when value is coerced

v0.5.0 parallel workflows on local host

miniwdl run

The built-in runner can now parallelize tasks and sub-workflows on the local host, using Docker Swarm locally to schedule containers (which it'll enable automatically). Start with miniwdl run_self_test to quickly test the installation viability.

Known limitations:

• Task output files must reside in or under the initial working directory (#214)
• Task input files are mounted read-only by default; commands to rename or remove them can succeed only with --copy-input-files (#210)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)
• Scheduler considers each tasks's runtime.cpu (default 1) versus the available host processors, but ignores runtime.{memory,disks} (#250)
• Scheduler may exceed the -@ parallelism limit argument when sub-workflows are called, potentially up to # host processors (#203).

Please vote on these and/or report other interoperability problems via Issues!

Other changes:

• Add :latest tag to runtime.docker if no other tag present (#232 @jdidion)
• Mount all input files into one in-container directory (except when filenames would collide) to improve interoperability using hts index files

miniwdl cromwell

• Cromwell 46.1

miniwdl check

• Suppress UnusedImport warning when imported struct definitions are used (#236 @DavyCats)

WDL

• Fix str() on Apply AST nodes for certain functions (#242 @DavyCats)

v0.4.1 maintenance

miniwdl run

• reduce likelihood of CPython RecursionError due to deep-copying of WDL values (#239 @nh13)

Standing limitations:

• Calls run sequentially (#202, #203)
• Task input files are mounted read-only; commands to rename or remove them will fail (#210)
• Task output files must reside in or under the initial working directory (#214)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)

Please vote on these and/or report other interoperability problems via Issues!

miniwdl cromwell

• accept relative file paths in input JSON file, interpreting them relative to current working directory (#230 @prihoda)
• Cromwell 45.1

WDL

• recognize version development and use the most up-to-date grammar available (#233 @jdidion)

v0.4.0

miniwdl run

• Improved logging with tty colors and other enhancements
• New subcommand miniwdl run_self_test runs a little workflow to test the installation
• Library functions involving file I/O (e.g. read_string(), write_lines()) are now available in workflows as well as tasks (#204)
• Prevent use of host files neither supplied in the workflow inputs nor generated by the workflow (e.g. /etc/passwd)
• Fix up type coercions in call inputs and struct instances

Remaining limitations:

• Calls run sequentially (#202, #203)
• Task input files are mounted read-only; commands to rename or remove them will fail (#210)
• Task output files must reside in or under the initial working directory (#214)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)

Please vote on these and/or report other interoperability problems via Issues!

WDL package

Breaking API changes:

• WDL.Env rewritten with an easier, more-Pythonic interface
• Simplify compound Value constructors so that e.g. Value.Array needs to be told the array item type rather than the full Type.Array(...)

We now plan to minimize further breaking API changes to the WDL AST data structures, on the way to a 1.0 version.

Other changes:

• Add __str__ to Expr classes (#215 @DavyCats)
• String-to-Int/Float coercions allowed, may fail at runtime (#198)
• Statically reject use of stdout(), stderr(), and glob() outside of task output sections
• Set pos : SourcePosition attribute on the subset of WDL.Type objects instantiated by the syntax parser (e.g. Decl.type & StructTypeDef @dinvlad)
• Fix WDL.resolve_file_import() pass-through call to WDL.Tree.resolve_file_import() (@dinvlad)
• Tighten name collision rules for 1.0+ workflow outputs (#196)

Housekeeping

• Parallelized unit test suite using pytest
• Cromwell 45, lark-parser 0.7.3

v0.3.0

miniwdl run

🏁 This is the first release with a credible built-in capability to execute workflows on the local host. It's in early testing, but please try it out and let us know about any interoperability problems via Issues!

Salient limitations:

• Calls run sequentially (#202, #203)
• Task input files are mounted read-only; commands to rename or remove them will fail (#210)
• Task output files must reside in or under the initial working directory (#214)
• Library functions that involve file I/O (e.g. read_string(), write_lines()) are available in tasks but not workflows (#204)
• Namespace discrepancies may interfere with specifying optional inputs to sub-workflow calls (#193)

miniwdl check

• detects cyclic dependencies involving scatter & conditional expressions

miniwdl cromwell

• accept --options with Cromwell workflow options JSON file (#206 @prihoda)
• fix CLI arg completer regression
• Cromwell 44

WDL

• Refactor workflow AST to facilitate runtime dependency analysis, with WorkflowNode and WorkflowSection base classes exposing dependency ID strings, and explicit Gather nodes
• Refactor WDL.load() to permit overriding file-reading logic (#177 @dinvlad)
• Rename WDL.{Workflow,Scatter,Conditional}.elements to .body
• Rename WDL.Expr.Ident.ctx to WDL.Expr.Ident.referee

Housekeeping

• Replace wget subprocess calls with Python standard library functionality (#191 @nh13)
• Make all of package's internal imports relative & clean up naming for readability

v0.2.2

miniwdl cromwell

• disable --path since it doesn't work with Cromwell (see #131, #118)

WDL

• clean up & document programmatic access to miniwdl check lint warnings
• parser: allow comma after last element of map and object literals, as well as arrays

miniwdl run

• new subcommand shares miniwdl cromwell's command-line interface, instead invoking the work-in-progress internal runner. Currently, it can be used to run individual tasks (WDL documents with one task) on the local host.

v0.2.1

miniwdl check

• UnknownRuntimeKey for task runtime entries not on a list of known keys (#150 @pshapiro4broad)

miniwdl cromwell

• accept --input INPUTS.json with a Cromwell-style input JSON file, merging in any additional inputs from the command line (#165 @prihoda)
• allow override of Cromwell JAR and config file through arguments and env vars (#171 @cmarkello)
• Cromwell 42

WDL package

• struct initialization from map literal (where the key type is String-coercible and the value type is coercible to each respective struct member type)
• struct literals (object/map) may omit optional struct members
• parser: remove remaining constraints on the order of workflow elements/sections (#173 @patmagee)

meta

• Add biowdl/aligning and biowdl/expression-quantification test corpi
• miniwdl package is available through conda-forge (@rhpvorderman)

v0.2.0

miniwdl check

• detect call name collisions even if callee has no outputs (#133)
• UnusedDeclaration ignore likely secondary files based on name suffix case-insensitively

miniwdl cromwell

• adds support for shell tab-completion of the available input names, using argcomplete

WDL

• add WDL.values_to_json and WDL.values_from_json for conversion between WDL.Env.Values and Cromwell-style JSON
• WDL.load can take source_text : str of the source code instead of reading it from disk/URI (#154)
• provide pos : SourcePosition for SyntaxError and ImportError (#153 @dinvlad)
• task 'non-input' declarations may interleave anywhere among other task sections (#156)
• workflow output expressions can reference previous workflow outputs (#155 @rhpvorderman)
• exclude output declarations from Workflow.available_inputs (#135 @patmagee)
• StdLib refactoring and several additional function implementations
• Address value coercion loose ends: enforcing optional/nonempty quantifiers, recursive coercions in compound types, refined error objects & messages
• Errors no longer embed source line/column in the textual message; the pos : SourcePosition attribute remains available

meta

• Add biowdl/tasks test corpus
• Improving test suite Mac OS X compatibility although some issues remain

Starting from this release, work-in-progress WDL runner functionality is included in the code base & test suite; but it's still far from usable for general purposes.