Separate entrypoints and standalone scripts (#285)

* Initialize MLflow environment variable for filesystem tracking

* Make extract clips a uv-runnable standalone script

* Fix create zarr dataset guide

* Create zarr module

* Group visualization scripts

* Rename

* Update references

* Add other scripts and README

* Remove old utils. Remove entrypoint for extract clips

Author: sfmig

MANIFEST.in

@@ -8,6 +8,7 @@ recursive-include bash_scripts *.sh
 recursive-include notebooks *.py
 recursive-include notebooks *.ipynb
 recursive-include scripts *.py
+recursive-include scripts *.md
 recursive-include crabs *.yaml
 recursive-include guides *.png
 

bash_scripts/run_extract_loop_clips_array.sh

@@ -61,33 +61,26 @@ if [[ $SLURM_ARRAY_TASK_COUNT -ne $NUM_CSV_ROWS ]]; then
 fi
 
 # -----------------------------
-# Create virtual environment
+# Set up uv
 # -----------------------------
-# TODO: replace with uv
-module load miniconda
+# extract_loop_clips.py is a standalone (PEP 723) script: uv fetches it
+# from the repository, resolves its inline dependencies into an ephemeral
+# environment, and runs it. No package install or virtual environment is
+# needed.
+module load uv
+
+# set uv cache dir to /ceph/scratch/sminano
+# (should be faster than the home directory cache and gets purged regularly)
+export UV_CACHE_DIR=/ceph/scratch/sminano/uv-cache
+# copy (instead of symlink) files across filesystems (ceph cache vs tmpfs)
+export UV_LINK_MODE=copy
+export UV_HTTP_TIMEOUT=120  # seconds
+
+# Remote URL of the standalone script for the selected branch
+SCRIPT_URL="https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/$GIT_BRANCH/scripts/extract_loop_clips.py"
 
-ENV_NAME=crabs-extract-$SLURM_ARRAY_JOB_ID-$SLURM_ARRAY_TASK_ID
-ENV_PREFIX=$TMPDIR/$ENV_NAME
-
-conda create \
-    --prefix $ENV_PREFIX \
-    -y \
-    python=3.12
-
-# activate environment
-source activate $ENV_PREFIX
-
-# install crabs package in virtual env
-python -m pip install git+https://github.com/SainsburyWellcomeCentre/crabs-exploration.git@$GIT_BRANCH
-
-# log pip and python locations
-echo $ENV_PREFIX
-which python
-which pip
-
-# print the version of crabs package (last number is the commit hash)
 echo "Git branch: $GIT_BRANCH"
-conda list crabs
+echo "Script: $SCRIPT_URL"
 echo "-----"
 
 # ---------------------------------------
@@ -103,7 +96,7 @@ fi
 # -------------------------
 # Run extraction script
 # -------------------------
-extract-loops \
+uv run "$SCRIPT_URL" \
     --csv_filepath $CSV_PATH \
     --input_dir $INPUT_DIR \
     --output_dir $OUTPUT_DIR \
@@ -114,12 +107,6 @@ extract-loops \
 echo "Completed extraction of clip with task ID = $SLURM_ARRAY_TASK_ID"
 echo "--------------------------------------------------------"
 
-# -----------------------------
-# Cleanup
-# ----------------------------
-conda deactivate
-conda remove --prefix $ENV_PREFIX --all -y
-
 # ------------------
 # Copy logs to LOG_DIR
 # -------------------

crabs/utils/__init__.py crabs/zarr/__init__.pycrabs/utils/__init__.py renamed to crabs/zarr/__init__.py

@@ -25,9 +25,9 @@
 
 3.  **Download the create-zarr-dataset bash script from the 🦀 repository**
 
-    To do so, run the following command, which will download a bash script called `run_zarr_dataset_creation.sh` to the current working directory.
+    To do so, run the following command, which will download a bash script called `run_zarr_dataset.sh` to the current working directory.
     ```
-    curl https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/main/bash_scripts/run_zarr_dataset_creation.sh > run_zarr_dataset_creation.sh
+    curl https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/main/bash_scripts/run_zarr_dataset.sh > run_zarr_dataset.sh
     ```
 
     This bash script launches a SLURM array job to create a zarr dataset from a set of input VIA track files. Each job in the array processes files from a single video. With the command above, the version of the bash script downloaded is the one at the tip of the `main` branch in the [🦀 repository](https://github.com/SainsburyWellcomeCentre/crabs-exploration).
@@ -38,11 +38,11 @@
 >
 > - For example, to download the version of the file at the tip of a branch called `<BRANCH-NAME>`, edit the path above to replace `main` with `<BRANCH-NAME>`:
 >   ```
->   https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/<BRANCH-NAME>/bash_scripts/run_zarr_dataset_creation.sh
+>   https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/<BRANCH-NAME>/bash_scripts/run_zarr_dataset.sh
 >   ```
 > - To download the version of the file of a specific commit, replace `main` with `blob/<COMMIT-HASH>`:
 >   ```
->   https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/blob/<COMMIT-HASH>/bash_scripts/run_zarr_dataset_creation.sh
+>   https://raw.githubusercontent.com/SainsburyWellcomeCentre/crabs-exploration/blob/<COMMIT-HASH>/bash_scripts/run_zarr_dataset.sh
 >   ```
 
 4. **Edit the bash script if required**
@@ -63,7 +63,7 @@
     To launch a job, use the `sbatch` command with the path to the bash script:
 
     ```
-    sbatch path/to/run_zarr_dataset_creation.sh
+    sbatch path/to/run_zarr_dataset.sh
     ```
 
 6.  **Check the status of the job**
@@ -97,7 +97,7 @@ Sometimes some of the jobs in the array job fail due to non reproducible issues
     Run the edited bash script, to create a zarr dataset for the previously failed jobs:
 
     ```bash
-    sbatch path/to/edited/run_zarr_dataset_creation.sh
+    sbatch path/to/edited/run_zarr_dataset.sh
     ```
 
     If the array job runs successfully, a new zarr store (that we will call `store_2` here) will be generated.

crabs/utils/create_zarr_dataset.py crabs/zarr/create_dataset.pycrabs/utils/create_zarr_dataset.py renamed to crabs/zarr/create_dataset.py

@@ -54,7 +54,7 @@
     - `CSV_PATH`: path to the input csv file.
     - `INPUT_DIR`: path to the input directory containing the input videos.
     - `OUTPUT_DIR`: path to the output directory for the extracted loop clips.
-    - `GIT_BRANCH`: version of the 🦀 package to use. Usually we will use the version at the tip of the `main` branch.
+    - `GIT_BRANCH`: version of the standalone extraction script to fetch and run. Usually we will use the version at the tip of the `main` branch.
     - `VERIFY_FRAMES`: whether to verify frame count of the extracted clips matches the value in the csv file.
 
 

guides/CreateZarrDatasetForTracks.md

@@ -75,8 +75,7 @@ train-detector = "crabs.detector.train_model:app_wrapper"
 evaluate-detector = "crabs.detector.evaluate_model:app_wrapper"
 detect-and-track-video = "crabs.tracker.track_video:app_wrapper"
 # support utils
-extract-loops = "crabs.utils.extract_loop_clips:app_wrapper"
-create-zarr-dataset = "crabs.utils.create_zarr_dataset:app_wrapper"
+create-zarr-dataset = "crabs.zarr.create_dataset:app_wrapper"
 
 [build-system]
 requires = ["setuptools>=77", "wheel", "setuptools_scm[toml]>=8"]

guides/ExtractLoopClipsCluster.md

@@ -0,0 +1,3 @@
+Standalone scripts to support one-off tasks.
+
+They should be runnable with `uv`.

pyproject.toml

@@ -1,4 +1,16 @@
-"""Extract loop clips from input videos using ffmpeg."""
+"""Extract loop clips from input videos using ffmpeg.
+
+Standalone script: run with ``uv run scripts/extract_loop_clips.py ...``
+(uv installs the inline dependencies below into an ephemeral env). The
+``ffmpeg``/``ffprobe`` binaries must be available on PATH.
+"""
+
+# /// script
+# requires-python = ">=3.11"
+# dependencies = [
+#     "pandas",
+# ]
+# ///
 
 import argparse
 import subprocess