JupyterHub fails untidily when a home directory is unavailable, with unhelpful error message

[nxg]

Bug description

If a user attempts to start a server when their (NFS-mounted) home directory is unavailable, the resulting error message and log message are very unhelpful, to both the user and to the administrator.

Expected behaviour

Error message to user, and log message for admins, saying something like 'I can't find that home directory', or at least something which suggests it's an account or filesystem problem rather than a jupyterhub problem.

Actual behaviour

A user attempts to log in and start a server, but their home directory is unavailable. The user sees an error message:

Spawn failed: pid

What appears in the log is:

[E 2020-09-07 18:06:22.657 JupyterHub user:638] Unhandled error starting 2130509g's server: 'pid'
[E 2020-09-07 18:06:22.674 JupyterHub gen:598] Exception in Future <Task finished name='Task-441179' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, d
Traceback (most recent call last):
File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/tornado/gen.py", line 593, in error_callback
future.result()
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 852, in finish_user_spawn
await spawn_future
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 656, in spawn
raise e
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 560, in spawn
url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/sudospawner/spawner.py", line 101, in start
self.pid = reply['pid']
KeyError: 'pid'

As you can see, this is an unconventional spelling of 'I can't find the home directory'.

Perhaps that KeyError could be caught, and translated into the underlying cause, or something pointing in the right direction.

How to reproduce

I haven't verified the following steps, but based on my observations, and what I did to fix this locally, the following should exhibit this behaviour.

1. Create a user with a malformed home directory, or otherwise mangle the home directory -- perhaps merely changing the permissions would do it
2. Log in and wait for the server to be started.
3. ...that should show the error

Your personal set up

This user had an NFS-mounted home directory, but the automount map was wrong, so the home directory couldn't be mounted, and failed after a 10s (or thereabouts) timeout.

When I fixed the automount map, and verified that ls /home/<user> now worked, I was able to restart the server as admin, and the user was able to log in as normal.

Server OS: CentOS7
Client OS: don't know
JupyterHub version: 1.1.0

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[manics]

@nxg this looks like an issue with the spawner, so I've transferred the issue to the sudospawner repository.