Bug description

Expected behaviour

Pod should start, quickly.

Actual behaviour

Pod doesn't start, takes 5 minutes to timeout, then starts quickly the second time. Seems to be pod-reflector is getting restarted.

How to reproduce

1. Using binderhub.
2. Start a binder.
3. Container either already exists or is built fine.
4. Container starts, pod exists in cluster but hub can't seem to find it.
5. Wait 5 minutes, times out, complains something about pod-reflector among other things.
6. Pod starts in seconds.

Your personal set up

• Binderhub on a bare metal cluster.
• Not sure what details are relavent, ask away, I'm happy to give anything not in a k8s secret.
• Helm binderhub version binderhub-0.2.0-n870.h3d61c7a. I believe this to be a jupyterhub problem as it's failing in the hub pod it seems, but don't quite know how to get the associated jupyterhub version.

• Logs

# paste relevant logs here, if any

[W 2022-02-23 21:44:16.404 JupyterHub spawner:2861] Ignoring unrecognized KubeSpawner user_options: binder_launch_host, binder_persistent_request, binder_ref_url, binder_request, image, repo_url, token
[I 2022-02-23 21:44:16.417 JupyterHub log:189] 202 POST /hub/api/users/alex_orange-link-budget-binder-8iaji54m/servers/ (binder@192.168.132.3) 104.34ms
[I 2022-02-23 21:44:16.417 JupyterHub spawner:2302] Attempting to create pod jupyter-alex-5forange-2dlink-2dbudget-2dbinder-2d8iaji54m, with timeout 3
[E 2022-02-23 21:44:34.148 JupyterHub spawner:2581] Pod binderhub/jupyter-alex-5forange-2dlink-2dbudget-2dbinder-2dajne11h7 never showed up in reflector, restarting pod reflector
Some really long message, too much for me to parse if secrets in it or not
[I 2022-02-23 21:44:34.168 JupyterHub reflector:275] watching for pods with label selector='component=singleuser-server' in namespace binderhub
[E 2022-02-23 21:44:34.171 JupyterHub gen:623] Exception in Future <Task finished name='Task-667053' coro=<KubeSpawner._start() done, defined at /usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py:2466> exception=TimeoutError('pod binderhub/jupyter-alex-5forange-2dlink-2dbudget-2dbinder-2dajne11h7 did not start in 300 seconds!')> after timeout
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
future.result()
File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 2572, in _start
await exponential_backoff(
File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 184, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: pod binderhub/jupyter-alex-5forange-2dlink-2dbudget-2dbinder-2dajne11h7 did not start in 300 seconds!

[W 2022-02-23 21:44:38.789 JupyterHub _version:41] Single-user server has no version header, which means it is likely < 0.8. Expected 1.4.2
[I 2022-02-23 21:44:38.789 JupyterHub base:909] User alex_orange-link-budget-binder-8iaji54m took 22.450 seconds to start
[I 2022-02-23 21:44:38.789 JupyterHub proxy:285] Adding user alex_orange-link-budget-binder-8iaji54m to proxy /user/alex_orange-link-budget-binder-8iaji54m/ => http://x.x.x.x:8888
[I 2022-02-23 21:44:38.792 JupyterHub users:677] Server alex_orange-link-budget-binder-8iaji54m is ready
[I 2022-02-23 21:44:38.793 JupyterHub log:189] 200 GET /hub/api/users/alex_orange-link-budget-binder-8iaji54m/server/progress (binder@y.y.y.y) 22355.43ms