Skip to content

JH Hub pod crashes with OOM when there are too many kubernetes events in the same namespace #913

Description

@Freia3

Bug description

Hub pod crashes with OOM when there are too many kubernetes events in the same namespace. Issues appear starting from 35K event lines in the namespace the hub pod is running.
Setting c.KubeSpawner.events_enabled = False resolves the issue, but this is not preferable for us.

How to reproduce

Create at least 35K kubernetes event lines in the same namespace (kubect get events | wc -l). Then log into jupyterhub and watch hub pod crash.
We use this for the resources of the hub pod:

    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 50m
        memory: 50Mi

We prefer not to upgrade the resources.

  1. Go to hub log in
  2. Click on "start server"
  3. User pod will not start up
  4. Hub pod crashes with Out Of Memory error

Expected behaviour

Hub pod does not consume enormous amount of resources

Actual behaviour

Hub pod consumes up to 1.5GB of memory

Your personal set up

Using zero-to-jupyterhub-k8s helm chart version 4.2.0
jupyterhub-kubespawner==7.0.0
c.KubeSpawner.events_enabled = True

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions