feat: cross-environment GPU metrics parity (K8s, SLURM, Flux)

[pmady]

Summary

Enable consistent GPU metrics collection across Kubernetes, SLURM, and Flux environments using the same binary and output format. Users should be able to compare GPU performance across on-prem and cloud with identical metrics.

Motivation

HPSF TAC reviewer feedback: "what would be nice to have is a means to collect metrics in the same way across environments. Allow me to run a workload on-premises and in cloud and use the same tool to compare the GPU performance."

Tasks

• Add --env flag to auto-detect environment (k8s, slurm, flux, standalone)
• Include environment metadata in output (orchestrator, node, job ID)
• Unified JSON schema across all environments
• Comparison documentation with examples
• Contact NVIDIA re: go-nvml collaboration for cross-platform metrics

References

• Standalone CLI: cmd/gpu-metrics/main.go
• KEDA scaler: cmd/keda-gpu-scaler/main.go

[pmady]

hey @venkata22a — this one's yours if you want it. the main thing is a unified JSON envelope so output looks the same whether you're on k8s, slurm, flux, or bare metal. rough idea:

{
  "environment": {
    "type": "slurm",
    "hostname": "gpu-node-01",
    "collected_at": "2026-06-15T10:00:00Z"
  },
  "scheduler": { ... },
  "devices": [ ... ]
}

needs:

• new pkg/environ/ package that auto-detects the env (check for KUBERNETES_SERVICE_HOST, SLURM_JOB_ID, FLUX_JOB_ID, fall back to standalone)
• --env flag on gpu-metrics (auto/k8s/slurm/flux/standalone)
• always include hostname + timestamp in output
• keep it minimal, no big restructure needed

lmk if you have questions, happy to hop on a call