Support JupyterLab>=4.0

[bollwyvl]

Problem

One of the larger breaking changes from JupyterLab 4.x will apparently be landing on the 3.6 line, namely the frontend expecting a jupyter_server_ydoc-compatible WebSocket server.

This should provide missing identity features, and fixes for total-data-loss bugs when working with many users in RTC.

Proposed Solution

NB: I really don't know what I don't know about everything happening here, and would greatly appreciate any help from others more knowledgable 😍.

Ideally, implementing this change should still result in a lightweight, low-config labextension (and likely liteserverextension) that only requires a public or self-hostable (#4), signaling-only server process.

LiteServerExtension

Provide the same features as the https://github.com/jupyter-server/jupyter_server_ydoc/blob/v0.1.12/jupyter_server_ydoc/ydoc.py#L86.

For a full JS solution, it would be a relatively deep stack including

• dependency on ywasm (~900kb)
• some bespoke models with behavior as well.

Ultimately, it may still require a source of truth, a leader election process, or something, which may not be provided by a "naive" public yjs signaling server. Keeping this property may require an even more exotic implementation when it's all clients.

JupyterFrontEndPlugin

Update the provider (and related features) to match jupyterlab/jupyterlab#13222 (or whatever ends up being the final state).

Alternatives

LiteServerExtension

• use the absolute lightest-weight python jupyter_server_ydoc-compatible implementation
  • if this didn't require replacing the entire jupyterlite server (contents, etc) a la Use jupyverse as a "real" server jupyterlite#779
    • this could be feasible if even viewing a file isn't blocked by the (much larger than ywasm) emscripten machine starting up

Additional context

• it's possible this may trigger an abandoning of this repo altogether
• if this repo still has value, will probably require turning this into a monorepo
• probably blocking Update to JupyterLab 4 and Notebook 7 jupyterlite#826
• announced in JupyterLab 3.6.0 Release jupyterlab/jupyterlab#13221

[jtpio]

Renamed the issue to "Support JupyterLab>=4.0" since we are skipping 3.6.

[jtpio]

Also the new / updated extension will probably start from scratch anyway, as the upstream RTC implementation has changed drastically, and there is no notion of document providers anymore.

Which means the implementation could be done in a separate repo and this one be archived.

[westurner]

• src: https://github.com/jupyterlite/jupyterlab-webrtc-docprovider/blob/main/src/plugin.ts
• docs: https://jupyterlite.readthedocs.io/en/latest/howto/configure/rtc.html
• /? JupyterFrontEndPlugin: https://www.google.com/search?q=%22JupyterFrontEndPlugin%22
• https://github.com/QuantStack/yjs-widgets
  • https://github.com/QuantStack/yjs-widgets/blob/main/src/notebookrenderer/index.ts
    JupyterFrontEndPlugin<IJupyterYWidgetManager>
• https://github.com/search?q=IDocumentProviderFactory&type=code
  • https://jupyterlab.readthedocs.io/en/4.0.x/api/types/application.JupyterFrontEndPlugin.html
  • https://github.com/jupyterlab/jupyterlab/blob/main/docs/source/extension/extension_migration.rst#jupyterlab-3x-to-4x ; JupyterLab 3.x to 4.x
  • https://github.com/jupyterlab/extension-examples/tree/main
• offline in-browser version history; SQLite and/or IndexedDb
  • /? time slider: https://www.google.com/search?q=y-js+time+slider
  • /? yjs version history: https://www.google.com/search?q=yjs+version+history
  • https://github.com/yjs/yjs#overview #providers #bindings
  • https://github.com/yjs/y-indexeddb/blob/master/tests/y-indexeddb.tests.js
  • (IPython terminal history already logs to sqlite)
  • https://github.com/malte-j/y-op-sqlite
  • https://github.com/malte-j/y-op-sqlite/blob/main/src/index.ts
  • https://discuss.yjs.dev/t/correct-way-to-implement-version-history-like-google-doc/1691/4
  • https://discuss.yjs.dev/t/for-versioning-should-i-store-snapshot-or-document-copies/2421
  • https://discuss.yjs.dev/t/how-to-recover-to-the-specified-version/2301
  • https://github.com/yjs/y-prosemirror
  • https://github.com/ProseMirror/prosemirror-history/blob/master/test/test-history.ts
  • https://github.com/ProseMirror/prosemirror-history/blob/master/src/history.ts
  • https://github.com/yjs/yjs-demos/tree/main/prosemirror-versions ::

    In this variation of the ProseMirror demo, we allow you to create versions and render the differences and highlight the changes by a unique user-color. When you enable "Live Tracking", your editor becomes unresponsive and you can see diff created from edits from other users happening in real-time. Click on a created snapshot to render the differences to the previous snapshot.

    • nbdime diffs notebooks; does it already work in JupyterLite?

    The snapshot API is very versatile and allows you to render the differences between any two snapshots. In this examples, we simply created a linear version history. But using the same API you can model branching & merging as well.

    • So, another DAG with merges. Which could work with jujutsu (jj),
    • which supports git backends: https://github.com/martinvonz/jj#compatible-with-git (libgit2 + gitoxide (rust), native storage, )
      • https://github.com/martinvonz/jj/blob/main/lib/src/backend.rs#L384 ::
        pub trait Backend: Send + Sync + Debug
        • https://github.com/martinvonz/jj/blob/main/lib/src/local_backend.rs
        • https://github.com/martinvonz/jj/blob/main/lib/src/git_backend.rs
• and then identity, authentication, and authorization
  • W3C DIDs pk/sk pairs can be generated and registered or optionally centrally generated and then used to sign JSON
  • W3C DID supports GPG keys, _ keys, with a JSON-LD (RDF) schema
  • W3C DID Use Cases: https://www.w3.org/TR/did-use-cases/#uc
  • https://news.ycombinator.com/item?id=28814802 notes re: distributed identity and WoT
  • blockcerts
    • https://github.com/blockchain-certificates/cert-verifier-js#verify-a-blockcert-certificate
    • https://github.com/blockchain-certificates/trustdidweb-ts/blob/main/test/features.test.ts#L24-L30
  • sigstore
    • https://github.com/sigstore/sigstore-python#example-uses ; #verifying-against-a-signature-and-certificate
    • You can sign [python] packages with sigstore, which supports:
      • JWT tokens
      • OIDC OpenID Connect
  • https://w3c-ccg.github.io/universal-wallet-interop-spec/#Credential #Key
    • Add sigstore.dev examples w3c-ccg/universal-wallet-interop-spec#60
  • TLDR to sign notebooks with self-generate-able keys at: collaborative changeset with vector clock time, git commit time and git tag time, [python] package release containing a notebook built on a bicycle built for n release time