Skip to content

[FEAT]: Make it easier to use MIG GPUs with a --vram argument to cluv submit #62

Description

@lebrice

Context

A good fraction of the Fir / Rorqual / Nibi clusters is using MIG GPUs.
They are under-utilized.

When a user doesn't need an entire H100 GPU's vram, and is not doing distributed training (not using more than one gpu), then it would make a lot of sense to request MIG gpus on those clusters.

Proposed solution

cluv submit rorqual scripts/job.sh --vram=10GB

Also, the MIG slices should ideally not be hard-coded into cluv, but determined using the command described here: https://docs.alliancecan.ca/wiki/Multi-Instance_GPU

info -o "%G"|grep gpu|sed 's/gpu://g'|sed 's/),/\n/g'|cut -d: -f1|sort|uniq

Additional context

Currently, on these clusters, you don't get MIG unless you specifically ask for one (need to double-check if using --gpus=1 might get you a MIG slice, but I'm pretty sure it does not).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions