Context
A good fraction of the Fir / Rorqual / Nibi clusters is using MIG GPUs.
They are under-utilized.
When a user doesn't need an entire H100 GPU's vram, and is not doing distributed training (not using more than one gpu), then it would make a lot of sense to request MIG gpus on those clusters.
Proposed solution
cluv submit rorqual scripts/job.sh --vram=10GB
Also, the MIG slices should ideally not be hard-coded into cluv, but determined using the command described here: https://docs.alliancecan.ca/wiki/Multi-Instance_GPU
info -o "%G"|grep gpu|sed 's/gpu://g'|sed 's/),/\n/g'|cut -d: -f1|sort|uniq
Additional context
Currently, on these clusters, you don't get MIG unless you specifically ask for one (need to double-check if using --gpus=1 might get you a MIG slice, but I'm pretty sure it does not).
Context
A good fraction of the Fir / Rorqual / Nibi clusters is using MIG GPUs.
They are under-utilized.
When a user doesn't need an entire H100 GPU's vram, and is not doing distributed training (not using more than one gpu), then it would make a lot of sense to request MIG gpus on those clusters.
Proposed solution
Also, the MIG slices should ideally not be hard-coded into cluv, but determined using the command described here: https://docs.alliancecan.ca/wiki/Multi-Instance_GPU
Additional context
Currently, on these clusters, you don't get MIG unless you specifically ask for one (need to double-check if using --gpus=1 might get you a MIG slice, but I'm pretty sure it does not).