internal_ssl support

[minrk]

Proposed change

I don't think the proxy recognizes the internal ssl config of JupyterHub, so it will not work if internal ssl is configured.

Alternative options

Leave this feature unimplemented

Who would use this feature?

Deployments that want to use jupyterhub's internal SSL feature to encrypt intra-hub communcations

[kreuzert]

I'm currently looking into this topic. I'll summarize what I've learned so far about this and the (minimal) required changes for the Proxy class to support this in theory. I've only looked into the TraefikRedisProxy, other subclasses might be easier to integrate but are less interesting for a Z2JH K8s setup.

tl;dr:
TraefikProxy._dynamic_config_for_route(self, routespec, target, data) lacks a configurable option to add serversTransports.
This serversTransports is not supported by Traefiks RedisProvider and must be configured externally via another TraefikProvider (e.g. File or KubernetesCRD).

---

Motivation

I'm looking into this topic and writing this post for 3 reasons:

1. Other people might stumble across this.
2. In our K8s Setup JupyterHub setup configurable-http-proxy still uses to much memory, even with the newest version 5.1 ( Image: quay.io/jupyterhub/configurable-http-proxy@sha256:eb10d5bf045c... ).

3. Personal interested to learn new things about traefik + redis

Setup

Requirements:

You need an existing internal-ssl folder. This must be accessible by JupyerHub and your traefik installation. In a Kubernetes setup this might be a Secret, for this example that's a folder on my notebook.

Redis

For a minimal test setup I'm using a local redis-server (running as systemd script on my local Debian).

# apt update && apt install -y redis-server

I've added a password to /etc/redis/redis.conf

requirepass <password>
masterauth <password>

and restarted the redis server ( systemctl restart redis-server )

Traefik

I'm using a local traefik installed in my virtualenv with pip install traefik-bin , version 3.5.3 right now.

Unfortunately, the Traefik Redis Provider does not support the configuration of ServersTransports. We need to combine multiple providers for this internal_ssl setup to work correctly.

In this example I'm using a File Provider. In a Kubernetes Setup this can also be a KubernetesCRD Provider (not tested yet).

For this provider I've created the file ~/traefik/dynamic/transport.yml:

http:                                                                                                                 
  ServersTransports:                                                                                                  
    myTransport:
      serverName: localhost
      certificates:
        - certfile: /home/ubuntu/internal-ssl/proxy-client/proxy-client.crt
          keyfile: /home/ubuntu/internal-ssl/proxy-client/proxy-client.key
      rootCAs:
        - /home/ubuntu/internal-ssl/hub-ca_trust.crt
      insecureSkipVerify: false
      disableHTTP2: false

traefik --entryPoints.traefik.address=:8080/tcp --entryPoints.web.address=:8000/tcp --entryPoints.websecure.address=:8443/tcp --api.dashboard=true --ping=true --log.level=INFO --api.insecure=true --providers.redis --providers.redis.endpoints=localhost:6379 --providers.redis.password=<password> --providers.redis.rootkey=traefik --providers.file --providers.file.directory=~/traefik/dynamic --providers.file.watch=true

JupyterHub

Setup in a virtualenv:
pip install jupyterhub jupyterlab jupyterhub-traefik-proxy[redis]

jupyterhub_config.py:

c.JupyterHub.internal_ssl = True
c.JupyterHub.log_level = "DEBUG"

from jupyterhub_traefik_proxy.redis import TraefikRedisProxy

class SSLTraefikRedisProxy(TraefikRedisProxy):
    serversTransport = "myTransport@file"

    def _dynamic_config_for_route(self, routespec, target, data):
        traefik_config, jupyterhub_config = super()._dynamic_config_for_route(
            routespec, target, data
        )
        if self.serversTransport:
            for service_alias in traefik_config.get("http", {}).get("services", {}).keys():
                if "loadBalancer" in traefik_config["http"]["services"][service_alias].keys():
                    traefik_config["http"]["services"][service_alias]["loadBalancer"]["serversTransport"] = self.serversTransport

        return traefik_config, jupyterhub_config

c.JupyterHub.proxy_class = SSLTraefikRedisProxy
c.SSLTraefikRedisProxy.redis_password = "<proxy>"
c.SSLTraefikRedisProxy.should_start = False
c.SSLTraefikRedisProxy.traefik_api_url = "http://localhost:8080"
c.SSLTraefikRedisProxy.traefik_api_entrypoint = "traefik"
c.SSLTraefikRedisProxy.traefik_entrypoint = "web"
c.SSLTraefikRedisProxy.enable_setup_dynamic_config = False


from jupyterhub.auth import DummyAuthenticator
c.JupyterHub.authenticator_class = DummyAuthenticator

from jupyterhub.spawner import LocalProcessSpawner
c.JupyterHub.spawner_class = LocalProcessSpawner

The crucial part is adding the serversTransport entry to http.services.<ServiceName>.loadBalancer. Since that's coming from a different provider than redis, we have to add the suffix "@file".

Conclusion of Test Setup

This setup allows us to run a JupyerHub with internal ssl and secure communication between JupyterHub components.
In this setup the Traefik Proxy is listening on http://127.0.0.1:8000 , so no https between User and Traefik with this example.
The communication between Traefik and JupyterHub uses SSL. The communication between Traefik and JupyterLab also uses SSL.

Negative aspects

The huge downside is you cannot provide the serversTransport configuration via Redis. That's simply not supported by the RedisProvider ( more ). Instead we need to rely on another Provider. This can be any provider that supports the serversTransports.
Possible options are (AFAIK) KubernetesCRD or FileProvider (as shown in this setup).

Positive aspects

A minimal change in the TraefikRedisProxy allowed me to set up a working example with Traefik, Redis, JupyerHub.

Outlook

Next tasks:
I'm planning to set up a Z2JH Setup behind an ingress-nginx setup. Traefik will replace chp in the end.
For this I will look into:

• Traefik listening on https://8443 with proxy-api.crt/key
• Transfer this setup into a minimal Z2JH installation. Probably using an additional K8s ServiceAccount to allow KubernetesCRD in other namespaces. This may allow me to run a singleton traefik + redis setup to support multiple JupyterHubs.

[minrk]

Thanks for looking into this! If you'd like to get as far as a PR, that would be hugely appreciated. I'm not sure when I'll get to this one.

internal ssl might need to go in the 'static' config that we have to set up before the 'dynamic' config that loads from the KV store.

[kreuzert]

Unfortunately, I couldn't make this work in a production environment (Kubernetes Cluster, one traefik deployment, one etcd deployment, one jupyterhub with internal ssl). While at first glance everything looks fine, we're hitting two types of SSL Errors, which we don't see with a nginx, chp + jupyterhub setup.

At the moment, I don't fully understand the reason for that. I assume traefik reuses connections that tornado has already closed.

The full setup + errors are posted here #278 . If someone wants to look into it further and has more experience with traefik / tornado / tls handshakes