Unable to respawn after restarting JupyterHub

[ryland-goldman]

Bug description

Whenever I use systemctl restart jupyterhub.service, I am unable to spawn a new server session. I get this message:

I am able to fix it by manually removing the file /run/jupyter-[username]-singleuser but I feel like there's something wrong with my configuration.

Your personal set up

• OS: Ubuntu 24.04 LTS
• Version: Python 3.12.3, Jupyter 5.1.0

Logs

Aug 24 19:20:25 myserver jupyterhub[790246]: [I 2024-08-24 19:20:25.712 JupyterHub proxy:752] Starting proxy @ http://127.0.0.1:8193
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.259 [ConfigProxy] info: Proxying http://127.0.0.1:8193 to (no default)
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.267 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.418 JupyterHub app:3690] Hub API listening on http://127.0.0.1:8081/hub/
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.418 [ConfigProxy] info: 200 GET /api/routes
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.421 [ConfigProxy] info: 200 GET /api/routes
Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.421 JupyterHub proxy:477] Adding route for Hub: / => http://127.0.0.1:8081
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.424 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.426 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081
Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.427 [ConfigProxy] info: 201 POST /api/routes/
Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.427 JupyterHub app:3731] JupyterHub is now running at http://127.0.0.1:8193
Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.807 JupyterHub provider:661] Creating oauth client jupyterhub-user-[username]
Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.842 JupyterHub user:1007] Unhandled error starting [username]'s server: [Errno 17] File exists: '/run/jupyter-[username]-singleuser'
Aug 24 19:20:26 myserver jupyterhub[790246]:     Traceback (most recent call last):
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "/[venv directory]/lib/python3.12/site-packages/jupyterhub/user.py", line 920, in spawn
Aug 24 19:20:26 myserver jupyterhub[790246]:         await asyncio.wait_for(f, timeout=spawner.start_timeout)
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "/usr/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
Aug 24 19:20:26 myserver jupyterhub[790246]:         return await fut
Aug 24 19:20:26 myserver jupyterhub[790246]:                ^^^^^^^^^
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemdspawner.py", line 331, in start
Aug 24 19:20:26 myserver jupyterhub[790246]:         await systemd.start_transient_service(
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemd.py", line 143, in start_transient_service
Aug 24 19:20:26 myserver jupyterhub[790246]:         environment_file = make_environment_file(
Aug 24 19:20:26 myserver jupyterhub[790246]:                            ^^^^^^^^^^^^^^^^^^^^^^
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemd.py", line 54, in make_environment_file
Aug 24 19:20:26 myserver jupyterhub[790246]:         ensure_environment_directory(environment_file_directory)
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemd.py", line 26, in ensure_environment_directory
Aug 24 19:20:26 myserver jupyterhub[790246]:         os.makedirs(environment_file_directory, mode=0o700, exist_ok=True)
Aug 24 19:20:26 myserver jupyterhub[790246]:       File "<frozen os>", line 225, in makedirs
Aug 24 19:20:26 myserver jupyterhub[790246]:     FileExistsError: [Errno 17] File exists: '/run/jupyter-[username]-singleuser'
Aug 24 19:20:26 myserver jupyterhub[790246]:     
Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.884 JupyterHub pages:312] Error starting server [username]: [Errno 17] File exists: '/run/jupyter-[username]-singleuser'
Aug 24 19:20:26 myserver jupyterhub[790246]:     Traceback (most recent call last):
Aug 24 19:20:26 myserver jupyterhub[790246]:     NoneType: None
Aug 24 19:20:26 myserver jupyterhub[790246]:     
Aug 24 19:20:26 myserver jupyterhub[790246]: [W 2024-08-24 19:20:26.884 JupyterHub web:1873] 500 GET /hub/spawn (127.0.0.1): Unhandled error starting server [username]
Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.922 JupyterHub log:184] {
Aug 24 19:20:26 myserver jupyterhub[790246]:       "X-Forwarded-Port": "80",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Connection": "Keep-Alive",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "X-Forwarded-Server": "[domain]",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "X-Forwarded-Host": "[domain]",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "X-Forwarded-For": "[ipaddress],127.0.0.1",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "X-Forwarded-Proto": "https,http",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Accept-Language": "en-GB,en;q=0.9",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Sec-Fetch-Dest": "document",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Referer": "https://[domain]/user/[username]/lab/tree/src/py/refinitiv/Notebooks/Untitled.ipynb",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Sec-Fetch-Mode": "navigate",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Accept-Encoding": "gzip, deflate, br",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Sec-Fetch-Site": "same-origin",
Aug 24 19:20:26 myserver jupyterhub[790246]:       "Host": "[domain]"
Aug 24 19:20:26 myserver jupyterhub[790246]:     }
Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.923 JupyterHub log:192] 500 GET /hub/spawn ([username]@127.0.0.1) 161.32ms
Aug 24 19:20:27 myserver jupyterhub[790246]: [I 2024-08-24 19:20:27.154 JupyterHub log:192] 302 GET /user/[username]/api/kernels?1724527226931 -> /hub/user/[username]/api/kernels?1724527226931 (@127.0.0.1) 0.77ms
Aug 24 19:20:27 myserver jupyterhub[790246]: [W 2024-08-24 19:20:27.244 JupyterHub base:1614] Failing suspected API request to not-running server: /hub/user/[username]/api/kernels
Aug 24 19:20:27 myserver jupyterhub[790246]: [W 2024-08-24 19:20:27.244 JupyterHub log:192] 424 GET /hub/user/[username]/api/kernels?1724527226931 ([username]@127.0.0.1) 5.67ms
Aug 24 19:20:27 myserver jupyterhub[790246]: [I 2024-08-24 19:20:27.421 JupyterHub log:192] 302 GET /user/[username]/api/sessions?1724527227371 -> /hub/user/[username]/api/sessions?1724527227371 (@127.0.0.1) 0.66ms
Aug 24 19:20:27 myserver jupyterhub[790246]: [W 2024-08-24 19:20:27.516 JupyterHub base:1614] Failing suspected API request to not-running server: /hub/user/[username]/api/sessions

File `/etc/systemd/system/jupyterhub.service`:

[Unit]
Description=JuputerHub

[Service]
Type=simple
ExecStart=/[venv directory]/jupyterhub --port=8193 -f /[config directory]/jupyterhub_config.py
User=root
Group=root

Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Configuration:

c = get_config()
c.JupyterHub.spawner_class = "systemd"
c.Spawner.notebook_dir = "/directory"
c.SystemdSpawner.extra_paths = ["[venv directory]/bin"]
c.SystemdSpawner.dynamic_users = True
c.SystemdSpawner.readwrite_paths = ["/directory"]
c.SystemdSpawner.disable_user_sudo = False
c.SystemdSpawner.isolate_devices = True
c.SystemdSpawner.user_workingdir = "/directory"
c.JupyterHub.bind_url = 'http://127.0.0.1:8193'

Full environment

alembic==1.13.2
annotated-types==0.7.0
anyio==4.4.0
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-lru==2.0.4
attrs==24.2.0
babel==2.16.0
beautifulsoup4==4.12.3
bleach==6.1.0
certifi==2024.7.4
certipy==0.1.3
cffi==1.17.0
chardet==3.0.4
charset-normalizer==3.3.2
comm==0.2.2
contourpy==1.2.1
cryptography==43.0.0
cycler==0.12.1
DateTime==5.5
debugpy==1.8.5
decorator==5.1.1
defusedxml==0.7.1
deprecation==2.1.0
docker==7.1.0
dockerspawner==13.0.0
escapism==1.0.1
eventemitter==0.2.0
executing==2.0.1
fastjsonschema==2.20.0
fonttools==4.53.1
fqdn==1.5.1
greenlet==3.0.3
h11==0.14.0
h2==3.2.0
hpack==3.0.0
httpcore==1.0.5
httpx==0.27.0
hyperframe==5.2.0
idna==2.10
ipykernel==6.29.5
ipympl==0.9.4
ipython==8.26.0
ipython-genutils==0.2.0
ipywidgets==8.1.5
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.4
json5==0.9.25
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.5
jupyter_client==8.6.2
jupyter_core==5.7.2
jupyter_server==2.14.2
jupyter_server_terminals==0.5.3
jupyterhub==5.1.0
jupyterhub-systemdspawner==1.0.1
jupyterlab==4.2.4
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
jupyterlab_widgets==3.0.13
kiwisolver==1.4.5
Mako==1.3.5
MarkupSafe==2.1.5
matplotlib==3.9.2
matplotlib-inline==0.1.7
mistune==3.0.2
nbclient==0.10.0
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
notebook==7.2.1
notebook_shim==0.2.4
numpy==2.1.0
oauthlib==3.2.2
overrides==7.7.0
packaging==24.1
pamela==1.2.0
pandas==2.2.2
pandocfilters==1.5.1
parso==0.8.4
pexpect==4.9.0
pillow==10.4.0
platformdirs==4.2.2
prometheus_client==0.20.0
prompt_toolkit==3.0.47
psutil==6.0.0
ptyprocess==0.7.0
pure_eval==0.2.3
pycparser==2.22
pydantic==2.8.2
pydantic_core==2.23.0
Pygments==2.18.0
PyMySQL==1.1.1
pyOpenSSL==24.2.1
pyparsing==3.1.2
python-configuration==0.12.1
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
pytz==2024.1
PyYAML==6.0.2
pyzmq==26.2.0
qtconsole==5.5.2
QtPy==2.4.1
referencing==0.35.1
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986==1.5.0
rfc3986-validator==0.1.1
rpds-py==0.20.0
scipy==1.14.1
Send2Trash==1.8.3
setuptools==73.0.1
six==1.16.0
sniffio==1.3.1
soupsieve==2.6
SQLAlchemy==2.0.32
stack-data==0.6.3
terminado==0.18.1
tinycss2==1.3.0
tornado==6.4.1
traitlets==5.14.3
types-python-dateutil==2.9.0.20240821
typing_extensions==4.12.2
tzdata==2024.1
uri-template==1.3.0
urllib3==2.2.2
validators==0.33.0
watchdog==4.0.2
wcwidth==0.2.13
webcolors==24.8.0
webencodings==0.5.1
websocket-client==1.8.0
widgetsnbextension==4.0.13
zope.interface==7.0.1

[harvidsen]

I have the same problem. i see

File "/nix/store/ng8wivb5xfpyygld64sf3a2w4xx2yaff-python3-3.12.5-env/lib/python3.12/site-packages/system>
         ensure_environment_directory(environment_file_directory)
       File "/nix/store/ng8wivb5xfpyygld64sf3a2w4xx2yaff-python3-3.12.5-env/lib/python3.12/site-packages/system>
         os.makedirs(environment_file_directory, mode=0o700, exist_ok=True)
       File "<frozen os>", line 225, in makedirs
     FileExistsError: [Errno 17] File exists: '/run/jupyter-harvidsen-singleuser'

when running on NixOS. Which seems weird since exist_ok=True. However, the files seems to be a symlink to somewhere i cannot find

sudo ls -la /run/jupyter-harvidsen-singleuser
lrwxrwxrwx 1 root root 36 Oct  9 12:15 /run/jupyter-harvidsen-singleuser -> private/jupyter-harvidsen-singleuser

So perhaps the issue lies in how the directory (/symlink) is created?

[liamdiprose]

@harvidsen The symlink works if your working directory is /var/lib/.

I can't find any mention of symlink behaviour in the documentation. I wonder if exist_ok=True is still checking for a valid directory and errors when it finds a broken symlink.

In the case of dynamic_users=True, would it be better to skip the ensure_environment_directory call and let systemd manage the runtime directory?

https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#DynamicUser=

Use RuntimeDirectory= (see below) in order to assign a writable runtime directory to a service, owned by the dynamic user/group and removed automatically when the unit is terminated.