When should we update Kubernetes / k3s server?

[rgaiacs]

Related to #3453

Information from https://kubernetes.io/releases/

Version	End of Life
1.32	2026-02-28
1.33	2026-06-28
1.34	2026-10-27

2i2c.mybinder.org is running Kubernetes 1.31 and gesis.mybinder.org is running Kubernetes 1.32.

[manics]

As long as we first test on staging I think it's fine to upgrade straight away. The main limitation is people's time to do it.

[minrk]

Should we enable automatic k3s updates? https://docs.k3s.io/upgrades/automated

[minrk]

I ran a manual update on staging and it worked fine, but it was only a patch release change because it was already on 1.33.

I can run an update on 2i2c. It appears to be just running our install k3s command on any given node to upgrade to current stable.

[minrk]

2i2c upgrade went fine as well, now on 1.33.6, same as staging. Want me to do GESIS as well? I can in the morning.

[minrk]

Didn't appear to be any interruption. Upgrade took about a minute, no deployment pods stopped or appeared to be disrupted. It took a little for some system pods to restart and recover (metrics-server), but they managed it without intervention.

[minrk]

I added auto-upgrade config in #3551 and applied it to bids-ovh and staging. I can apply it to GESIS and see how it fares, if you want, @rgaiacs

[rgaiacs]

Should we enable automatic k3s updates?

Looks good to me.

I can apply [the auto-upgrade config] to GESIS and see how it fares

Yes, please.

[minrk]

okay, I'll apply the auto-upgrade on GESIS with a schedule to run while I'm watching today, then move it to Sundays.

[manics]

It'd be nice if we could have staging auto-upgrade N days after a stable K3S release, and prod auto-upgrade N+7 days after the release- something for the future!

[minrk]

Definitely worth considering.

The scheduling only seems to be able to understand weekdays, so we can't really do that with just the schedule.

The alternative to subscribing to a channel with an upgrade window is to add the upgrade plan to CD and specify a version rather than a channel. Then upgrades can only happen on merged PRs, rather than on any kind of schedule.

We'd need to update our watch dependencies to get k3s versions, but that shouldn't be a big deal.

As it is now, all our k3s clusters have auto-upgrade enabled for Sunday 0-2 UTC, and all are running current stable 1.33.6. GESIS did the upgrade via the auto-upgrade config (I updated the schedule temporarily to run this morning while I was watching), while the others (hetzner-2i2c, staging, bids-ovh) were upgraded by the manual k3s install command.
While the upgrade plan is in this repo, it is not hooked up to CD, so applying it is manual.

[rgaiacs]

Thanks a lot @minrk for update the GESIS server. <3

[minrk]

I worked on my idea above last week, where we have explicit versions upgraded by PRs, instead of auto-upgrade. It's a bit of work, but not massive.

One option that might be better is to run staging one version ahead, so on the latest channel instead of stable. But rolling back if there's a failure is always going to be a pain, no matter what approach we take.

[manics]

I think @rgaiacs's work in #3158 means we're close to being able to deploy and test most of the stack on a GitHub runner.

[rgaiacs]

we're close to being able to deploy and test most of the stack on a GitHub runner.

Sorry. I'm a little lost here. What is the missing pieces?

GitHub hosted runners with Ubuntu 24.04 include

• kind 0.31.0
• Minikube 1.37.0