v1.15.0rc1

miniwdl v1.15.0 adds support for WDL 1.2. (version 1.2)

As this will be the largest miniwdl update in years, we're issuing at least this one prerelease version for broader testing & feedback (Note: rc0 skipped due to a packaging snafu). As a prerelease version, you must explicitly request it:

pip3 install miniwdl==1.15.0rc1

There should be no breaking changes in the Python API and CLI, but the runner has a few changes & restrictions that may necessitate ops/deploy adjustments, while advanced usage of internal (non-API) functionality might be more affected.

WDL 1.2 has many useful new featuress, of which we'll highlight just a few here:

Finalized development features

miniwdl's version development has supported these for some time, now finalized in WDL 1.2:

• Directory inputs and outputs
• Multi-line strings delimited by <<< and >>>
• env modifier for task declarations, loading input values into environment variables to use instead of WDL ~{interpolation}

Source-relative input paths

Task and workflow declarations may bind File and Directory declarations to hard-coded relative paths, which the runner resolves in the directory containing the WDL file. This allows helper scripts and reference data to live as separate files alongside the WDL source, instead of inlining them in task commands, taking them as workflow inputs, or baking them into docker images. (Those options each remain useful & appropriate in different cases.)

annotate/
├── annotate.wdl
├── annotate.py
└── gene_panel.tsv

version 1.2

task annotate {
    input {
        File variants
        # Defaults resolved next to annotate.wdl:
        File panel = "gene_panel.tsv"
        File script = "annotate.py"
    }
    command <<<
        python3 ~{script} --panel ~{panel} ~{variants} > annotated.tsv
    >>>
    output {
        File annotated = "annotated.tsv"
    }
}

# From anywhere on the system — the helper script and gene panel still resolve:
miniwdl run /opt/pipelines/annotate/annotate.wdl variants=sample.vcf

Relative paths are resolved this way in input declarations, their defaults, and private declarations — including inside compound values such as Array[File] and Map[File, …]. In task output sections, relative paths are still interpreted relative to the task's execution directory.

Notes:

• ⚠️ Isolation change. This feature slightly relaxes the runner's default filesystem access model. Previously, a run could only read local files and directories explicitly listed in its inputs (absent the [file_io] allow_any_input configuration setting). It can now also read files and directories in the WDL source tree where the relative path appears. (Consequently, if you override the [file_io] root configuration setting, miniwdl now requires the source tree as well as all input files to reside under that directory.)
• Call cache invalidation accounts for these paths: the runner records the source-relative paths a call used, so a cached result is invalidated if they change. This requires a new call-cache format version, invalidating all outputs cached before v1.15.0.
• miniwdl zip --add for adding non-WDL files now preserves their source-relative paths in the archive. For now (#878), the operator must explicitly --add all necessary source-relative files and directories (glob patterns are supported).
• Source-relative input paths are currently only supported for WDL source trees on a mounted filesystem, not when accessed by URI.

Other notable changes

• Path canonicalization: File/Directory values are lexically normalized (e.g. ./a//b → a/b, trailing slashes stripped from Directory).
• Optional File/Directory validation: an optional File?/Directory? input that exists but is the wrong kind (e.g. an actual directory for a File value) is now rejected rather than silently treated as absent (#888).
• Task retries: fixed input-path mount handling on task retry, which could cause missing inputs on retried attempts in some environments (#895).
• hints sections are accepted by the parser, but not yet used nor exposed through the API (#785).

---

This work was enabled by the WDL community's enormous efforts to draft and clarify the specification, as well as AI coding agents making it possible to ship with limited (human) resourcing. Mahalo!

v1.14.2

Getting Started

• Fix standard library bugs & compliance issues:
  • Float ops using integer division (#840 @aofarrel @lilymaryam)
  • collect_by_key() silently merging distinct Float keys with the same string representation
  • basename(path, "") returning empty string instead of filename
  • \r\n line end trimming in read_{lines,tsv,map}
  • Type inference issues in read_tsv(), zip(), and cross()
• Guard against out-of-range uid/gid in Docker Swarm backend (#839 @nh13)
• Dependency updates

NOTE: Skipped v1.14.{0,1} due to release process snafu's.

v1.13.1

This regular maintenance release addresses minor bugs and dependency issues.

• CLI: replace bullet with questionary (#777 @bshifaw)
• make JSON pair definitions case-insensitive (#789 @stxue1)
• fix asyncio.get_event_loop() error in newer Python versions (#791 #793 @wleepang)
• migrate pkg_resources usage to importlib_metadata (#794 #802 @hexylena)
• fix miniwdl configure --cfg FILENAME_IN_CWD.cfg (#805 @hexylena)
• ensure static type error for unknown fields in struct literals (#808 @gshiba)

Internal refactoring and implementation for WDL 1.2 continues in the background!

v1.13.0

v1.13.0 addresses several WDL 1.1 spec interoperability issues, with thanks to the reporting community members.

⚠️ These changes, while improving WDL spec compliance, bear slightly elevated risks of backwards-incompatibility with prior miniwdl versions. Please report any issues or unexpected behaviors for investigation.

Interoperability

• dedent task commands before eval/interpolation (#674 @adamnovak)
• allow float values for task runtime.cpu (#690 @adamnovak)
• round() "half up" instead of "to even" (#698 @stxue1)
• fix Array runtime typing issues (#699 #700 @stxue1)
• allow trailing commas in meta arrays (#708 @adthrasher)
• fix ValueError reading empty JSON into Map (#726 @ahvigil)
• refactor typechecking for equatability & comparability (#736)
• allow non-typesafe read_json(...) within arrays and if-then-else branches (#740 @ahvigil)

Runner

• allow namespace prefix in task-only input JSON files (#693 @stxue1)
• fix cache of task with empty outputs (#715 @adamnovak)
• fix recognition of exponential notation in JSON/YAML run input files (#717 @sach244n)
• --no-quant-check applies to comparison operators (#752 @stxue1)

Python WDL API

• Multi-line strings (development/1.2 feature) get first-class AST representation as WDL.Expr.MultiLineString, which is dedented upon evaluation (instead of during parsing into WDL.Expr.String).
• WDL.Type.* classes expose ty1.equatable(ty2) and ty1.comparable(ty2) operators, indicating whether it is valid to use the WDL == != or < <= > >= operators, respectively.

v1.13.0rc0

v1.13.0 addresses several WDL 1.1 spec interoperability issues, with thanks to the reporting community members.

Because these changes bear small backwards-compatibility risks, we begin with release candidate v1.13.0rc0 that must be expressly requested with: pip install miniwdl==1.13.0rc0. Please report any issues or unexpected behaviors for investigation over the next few weeks.

Interoperability

• dedent task commands before eval/interpolation (#674 @adamnovak)
• allow float values for task runtime.cpu (#690 @adamnovak)
• round() "half up" instead of "to even" (#698 @stxue1)
• fix Array runtime typing issues (#699 #700 @stxue1 )
• allow trailing commas in meta arrays (#708 @adthrasher)
• fix ValueError reading empty JSON into Map (#726 @ahvigil)
• refactor typechecking for equatability & comparability (#736)
• allow non-typesafe read_json(...) within arrays and if-then-else branches (#740 @ahvigil)

Runner

• allow namespace prefix in task-only input JSON files (#693 @stxue1)
• fix cache of task with empty outputs (#715 @adamnovak)
• fix recognition of exponential notation in JSON/YAML run input files (#717 @sach244n)

Python WDL API

• Multi-line strings (development/1.2 feature) get first-class AST representation as WDL.Expr.MultiLineString, which is dedented upon evaluation (instead of during parsing into WDL.Expr.String).
• WDL.Type.* classes expose ty1.equatable(ty2) and ty1.comparable(ty2) operators, indicating whether it is valid to use the WDL == != or < <= > >= operators, respectively.

v1.12.1

This maintenance release has no user-facing changes, but updates dependencies and modernizes the Python project tooling (#668) in preparation for the next development cycle.

v1.12.0

• Fix coercions of Array[File] and Array[Directory] to Array[String] (#669 @john-smith, #681 @markjschreiber)
• Ensure presence of optional fields in structs read from runner input JSON file (#686 @adthrasher)
• miniwdl run: accept CLI inputs of compound types (pairs, maps, structs) as single-quoted JSON; also accept Array[Directory]
• miniwdl check: warn when a document of an older WDL version imports a document of a newer WDL version
• Dependency updates

v1.11.1

Getting Started

Maintenance release:

• miniwdl input-template: add --no-namespace option to omit the top-level workflow name prefix
• fix download caching when config [file_io] root is used (miniwdl-ext/miniwdl-aws#20 @staskh)
• logging improvements and dependency updates

v1.11.0

v1.11.0

v1.10.0

• WDL spec compatibility:
  • == and != accept optional operands (#634 @jdidion)
  • fix resolution of imported struct type aliases when the original struct name is reused in current document (#635 @jdidion)
  • WDL 1.1: allow placeholder options like sep= in string expressions outside of task commands (#633 @adamnovak)
  • WDL 1.1: disable a few version development features that had been inadvertently enabled for version 1.1 (multi-line strings and optional call: keyword)
• allow extra JSON keys initializing structs (#524)
• fix bug in WDL.Env.merge() that could resurrect previously-shadowed bindings (#637 @adamnovak)
• miniwdl zip --additional to include any specified file(s) in the zip (#641 @rhpvorderman)