service_running doesn't account for chroot

[SomeoneSerge]

systemdspawner/systemdspawner/systemd.py

Line 158 in e75cadd

return ret == 0

systemctl is-active outputs a warning and exits with 0 when run in a chroot environment (by "design", to quote the designer), after which jupyterhub attempts to stop a non-existent unit and fails:

Running in chroot, ignoring command 'is-active'
Running in chroot, ignoring command 'stop'
500 GET /hub/spawn (127.0.0.1): Error in Authenticator.pre_spawn_start: Exception Could not stop already existing unit

[behrmann]

Maybe let's step back a bit and ask what it is you'd like to do? :)

The point Lennart makes in the source you link is quite valid, but - depending on what it is you wish to achieve - there's possible patches for systemdspawner. systemd itself provides ample features to make calls to chroot unnecessary (because it does it under the hood; RootDirectory= and friends) or the check above could be amended to also run systemd-detect-virt.

[SomeoneSerge]

If you so wish, but I must warn we'll be going off the point :) I want to run jupyterhub on NixOS and have access to user kernels (e.g. installed via ipykernel from venv or conda environments).

Now,

• There's no global /usr or /lib on NixOS, no global dynamic linker
• conda and binary distributions in pypi assume these globals paths
• -> they must be run either in a specially crafted namespaces (with fake global paths mounted) or with crafted environment variables
• otherwise things like import cv2 crash on dlopen()
• ipykernel only preserves the path to the python binary when generating the kernelspec
• systemdspawner attempts to spawn that executable without the required environment and, if I remember correctly, crashes already importing the ipykernel

The workaround I've been using is the one with the namespace (there's a convenience function in nix for that, called buildFHSUserEnv) and it works perfectly fine as long as I:

• Keep running jupyterhub as root
• Use LocalProcessSpawner

IIUC, this is rather orthogonal to RootDirectory=

[behrmann]

Oh, this is a very interesting problem and I've been eyeing Nix for Jupyterhub deployments, but haven't yet found the time to get my feet wet with Nix.

I don't know how buildFHSUserEnv jives with this, but isn't there also the problem that libraries in the store have explicitly set RPATHs pointing to other things in the store? I heard this makes using binary packages from other sources (conda, wheels from PyPI) - if not impossible - difficult?

Is my understanding, that currently you are using LocalProcessSpawner and pointing it at a chroot generated with buildFHSUserenv? If so that could, with a bit of hacking on systemdspawner, be moved over. You could then use RootDirectory= to use the same chroot.

Sorry if I'm misunderstanding your setup, my knowledge of Nix is rather rudimentary.

[SomeoneSerge]

@behrmann definitely nothing to be sorry about, and thank you for advancing the discussion:)

• You're right, Nix sets runpaths and dynamic linkers to absolute paths:

  $ patchelf --print-interpreter ~/.conda/bin/xz
  /lib64/ld-linux-x86-64.so.2
  $ patchelf --print-interpreter $(which xz)
  /nix/store/vjq3q7dq8vmc13c3py97v27qwizvq7fd-glibc-2.33-59/lib/ld-linux-x86-64.so.2

• This does mean that the executables in ~/.conda/bin will fail, because the linker they ask for doesn't exist
• One way to use them is to run them in namespaces, where /lib64&c are mounted on top of the host system.
  • It's a no-brainer to combine the nix packages you need into a tree you can mount/chroot into (e.g. using the mentioned buildFHSUserEnv)
  • ...and this is how the process actually spawned: https://github.com/NixOS/nixpkgs/blob/8c24a6741ba38ac199018b48ff4db262d1f8e30f/pkgs/build-support/build-fhs-userenv/chrootenv/chrootenv.c
• My set-up is that I run jupyterhub already in such a namespace. The proxy and the kernels just inherit it, along with the cgroup slice
• This currently breaks systemctl and setuid binaries

---

But to get back from nix and to the issue at hand, I'd be say to say that:

• systemctl is-running isn't ... -> bool, but ... -> Maybe bool
• service_running should
  • either use some other means of checking for running processes than directly running systemctl,
  • or have a policy for handling the failure,
  • or we just say that jupyterhub (in combination with systemdspawner at least) doesn't support chroot and alike

[behrmann]

I think I know understand your problem better and I learned something, thank you!

While I agree that something like Maybe bool would be a sensible signature for systemctl is-active, it's unfortunately by somewhat limited POSIX semantics (and backwards compatibility), so that won't change. It will probably be difficult and maybe prone to breakage trying to reimplement, although that would surely be possible.

I still think, that RootDirectory= would be the most promising avenue of solving your problem. Can buildFHSUserenv just provide the chroot without doing the actual chrooting via chrootenv? If so you could

• generate a FHS-compliant chroot for your Jupyterhub
• generate a service file for your Jupyterhub that uses systemdspawner and points RootDirectory= to the previously generated chroot. You can then get more external paths into the chroot via Bind*Paths= directives.

[behrmann]

I think the above would be the cleanest solution. If that is not possible there's other possible solutions. The environment variable SYSTEMD_IGNORE_CHROOT will short circuit the chroot test, but I don't know in what other problems you might run, e.g. no dbus in the chroot.

[SomeoneSerge]

Actually, I think d-bus and polkit still work (it's not quite chroot after all), at least I've just tried running d-feet from this sort of a namespace and it didn't fail

Thank you for SYSTEMD_IGNORE_CHROOT! I'll see what happens with that on