Add Temporal.io server to simcore stack (required by dynamic-scheduler)

[GitHK]

NOTES

• this issues is blocking ✨ Adds t_scheduler used for reliably running code in dynamic-scheduler ⚠️ osparc-simcore#9036 and will not be merged before it's fully addressed
• I ran an initial analysis to help us start a discussion about adding temporal.io to our OPS stack, since I don't have the final vision (but some partial information) I did some research and curated the output of what I think should work for us.
• Below suggestion was reviewed by me, while I can't guarantee all the code, the logic makes sense

---

NOTE: moved proposal here

Decided:

• ANE In the new PR description go though the New service requirements and evaluate against the swarm solution
• ANE script for CI to monitor that no workflows are running when PR is flagged
•  We go for the solution A with docker swarm and autos-setup image
•  We need to add a backup temporal database in Postgres

[GitHK]

Temporal.io Deployment Analysis for OPS Team

Context

The dynamic-scheduler service (in osparc-simcore) now requires a running Temporal server (temporal:7233).
Temporal reuses the existing PostgreSQL instance, creating two additional databases: temporal and temporal_visibility.
These do not conflict with the simcore application database (simcoredb).

This document compares two deployment paths from an operations perspective.
Both paths run the same Temporal server software and provide identical functionality to the application.

Repo references:

• osparc-simcore — github.com/ITISFoundation/osparc-simcore — the application repo. Defines the temporal and dynamic-schdlr service images and compose entries.
• osparc-ops-environments — github.com/ITISFoundation/osparc-ops-environments — this repo. Defines deploy constraints, resources, monitoring, and per-environment overrides.

---

Path A: Docker Compose (temporalio/auto-setup)

Uses the temporalio/auto-setup:1.29.1 image — an all-in-one container that:

• Checks if temporal and temporal_visibility databases exist in PostgreSQL
• Creates them and initializes schemas if missing
• Applies schema migrations if upgrading from a previous version
• Starts the Temporal server (all 4 roles: frontend, history, matching, internal-worker)

This is what the dev stack already uses in osparc-simcore/services/docker-compose.yml.

Pros (OPS perspective)

• Zero additional tooling — no Helm, no kubectl, no Kubernetes cluster required
• Matches existing infra — same Docker Swarm pattern as every other service in osparc-ops-environments
• Single container — one image, one service definition, no init jobs or sidecars
• Idempotent schema management — auto-setup creates/migrates schemas on every startup automatically; no manual migration steps
• Fast to deploy — add ~15 lines to docker-compose.yml.j2, set 4 env vars, done
• No new operational knowledge — managed identically to postgres, rabbit, redis in the ops repo
• Simple rollback — change the image tag back and redeploy

Cons (OPS perspective)

• Schema runs on every container start — adds 10-30 seconds to startup while it checks/applies schema. Not a problem in practice, but not as clean as a one-time Job
• No role separation — all 4 server roles run in one process. Cannot independently scale history vs matching vs frontend
• No built-in health check endpoint — must use TCP port check (nc -z localhost 7233) rather than an HTTP health endpoint
• Upgrade requires care — sequential minor version upgrades only (1.29 → 1.30 → 1.31, not 1.29 → 1.31). Schema auto-migration handles this per restart, but you must not skip versions
• Monitoring is DIY — Prometheus metrics on port 8000 exist, but no preconfigured dashboards; you add scrape targets manually
• Archived upstream examples — the temporalio/docker-compose repo is archived. New examples live in temporalio/samples-server/compose but Temporal's investment is clearly in the Helm chart

---

Path B: Kubernetes Helm Chart (temporalio/helm-charts)

Uses the official Temporal Helm chart (v1.1.1, actively maintained) which deploys:

• Separate pods for each server role (frontend, history, matching, worker)
• A schema setup Kubernetes Job (runs as Helm pre-install/pre-upgrade hook)
• Admin tools pod
• Temporal Web UI pod
• All configured via a single values.yaml

Pros (OPS perspective)

• First-class officially supported — this is Temporal's recommended production deployment method. Actively maintained (131 contributors, updated weekly)
• Proper schema management — database creation and schema migration run as a Kubernetes Job with Helm hooks (pre-install, pre-upgrade), not on every container start
• Role separation — frontend, history, matching, worker are separate Deployments. Each can be scaled independently
• Native secrets — existingSecret support, integrates with ExternalSecrets/SealedSecrets for GitOps
• Built-in monitoring — preconfigured Grafana dashboards for server-general, per-service metrics, SDK metrics
• GitOps ready — works with ArgoCD (Helm hooks → PreSync), Flux, Rancher, Terraform
• Upgrade path is cleaner — helm upgrade with automatic schema migration Job; rollback with helm rollback
• SSO for UI — built-in support for auth providers on the Temporal Web UI

Cons (OPS perspective)

• Requires a Kubernetes cluster — not available on all your deployment environments today (ANE's comment: did I recall this right?)
• New operational domain — Helm chart management, values.yaml tuning, K8s networking between simcore Swarm services and Temporal in K8s
• Cross-system networking — if simcore stays on Swarm but Temporal moves to K8s, you need network connectivity between the two (overlay network, ingress, or NodePort)
• More moving parts — 5+ pods instead of 1 container. More things to monitor, more potential failure points for a modest workload
• Over-engineered for current scale — role separation and independent scaling matter at thousands of workflows/sec. oSPARC's dynamic service scheduling is far below that threshold

---

Why Docker Swarm (Path A) — Recommended for Now

Your deployment infrastructure is Docker Swarm-based. The osparc-ops-environments repo deploys all services via:

1. Jinja2 templates (docker-compose.yml.j2) rendered with env vars from the config repo
2. Per-environment overlays (docker-compose.deploy.{aws,dalco,local,master,public}.yml) merged with yq
3. docker stack deploy --with-registry-auth -c <stack_file> $SIMCORE_STACK_NAME

Adding Temporal to this flow requires minimal changes and no new infrastructure. It deploys in the same stack as dynamic-schdlr, sharing the same overlay network — no cross-system networking issues. The auto-setup image handles everything that the Helm chart's schema Job would, just less elegantly.

Use this path when:

• K8s is not available in the target environment
• You want the fastest path to production
• The team's primary expertise is Docker Swarm
• Temporal workload is modest (dynamic service scheduling only)

Why Kubernetes (Path B) — Recommended for Future

The Helm chart is where Temporal invests engineering effort. Docker Compose examples are community-maintained and the original repo is archived. As your K8s infrastructure matures across environments, migrating Temporal to K8s gives you:

• Proper separation of concerns (schema migration as a Job, not a startup script)
• Independent scaling when workload grows
• Native integration with your K8s monitoring/secrets/GitOps stack

Use this path when:

• K8s cluster is available and operational in the target environment
• Team has Helm/K8s operational experience
• You want to align with Temporal's officially supported deployment method
• You anticipate growing Temporal workload beyond dynamic scheduling

---

Docker Swarm Deployment Suggestions

Service definition for docker-compose.yml.j2

Add to osparc-ops-environments/services/simcore/docker-compose.yml.j2 near the other infrastructure services:

  temporal:
    networks:
      - monitored
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
        order: stop-first
        failure_action: rollback
        delay: 10s
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
      placement:
        constraints:
          - node.labels.simcore==true
      resources:
        reservations:
          cpus: "0.1"
          memory: "256M"
        limits:
          cpus: "1.0"
          memory: "1G"

Environment variables

Add to the deployment config.
These are already defined in osparc-simcore/.env-devel and passed through to the dynamic-schdlr service.
They need to be present in the repo config file referenced by .config.location:

TEMPORALIO_HOST=temporal
TEMPORALIO_PORT=7233
TEMPORALIO_NAMESPACE=default
TEMPORALIO_TASK_QUEUE=dynamic-scheduler

Per-environment override to disable

For environments not yet ready for Temporal:

# docker-compose.deploy.{env}.yml
services:
  temporal:
    deploy:
      replicas: 0
  dynamic-schdlr:
    deploy:
      replicas: 0   # must also be disabled — it crashes without Temporal

Monitoring (optional)

Add Prometheus scrape labels:

  temporal:
    deploy:
      labels:
        - prometheus-job=temporal
        - prometheus-port=8000

Deployment order

No special ordering needed:

• auto-setup retries PostgreSQL connectivity internally
• dynamic-schdlr has a 120-second retry loop waiting for Temporal
• Docker Swarm restart policy handles transient failures

Upgrade procedure

1. Sequential minor versions only — do not skip (e.g. 1.29 → 1.30 → 1.31)
2. The image tag is defined in osparc-simcore/services/docker-compose.yml — it gets updated by the simcore dev team and flows into the generated stack file via prepare_simcore_stack.bash
3. auto-setup handles schema migration automatically on next deploy
4. Allow ~10 minutes for history shards to reload after upgrade
5. Optionally update temporalio/ui tag in osparc-ops-environments/services/simcore/docker-compose.yml.j2 (or in the ops compose if the UI is defined there)

Temporal UI

Already defined in osparc-simcore/services/docker-compose-ops.yml.
This flows into the ops stack. Current definition:

  temporal-ui:
    image: temporalio/ui:2.36.0
    environment:
      TEMPORAL_ADDRESS: temporal:7233
    ports:
      - "8233:8080"

---

Kubernetes Deployment Suggestions

Helm install

helm install temporal \
  --repo https://go.temporal.io/helm-charts \
  -f temporal-values.yaml \
  --namespace temporal \
  --create-namespace \
  --timeout 900s

Minimal temporal-values.yaml (PostgreSQL)

server:
  config:
    persistence:
      numHistoryShards: 512
      datastores:
        default:
          sql:
            createDatabase: true
            manageSchema: true
            pluginName: postgres12
            driverName: postgres12
            databaseName: temporal
            connectAddr: "<postgres-host>:5432"
            connectProtocol: tcp
            user: <postgres-user>
            existingSecret: temporal-db-secret
            secretKey: password
        visibility:
          sql:
            createDatabase: true
            manageSchema: true
            pluginName: postgres12
            driverName: postgres12
            databaseName: temporal_visibility
            connectAddr: "<postgres-host>:5432"
            connectProtocol: tcp
            user: <postgres-user>
            existingSecret: temporal-db-secret
            secretKey: password

web:
  enabled: true

Database secret

kubectl create secret generic temporal-db-secret \
  --namespace temporal \
  --from-literal=password=<postgres-password>

Or manage via ExternalSecrets/SealedSecrets for GitOps.

Networking to simcore (Swarm)

If simcore remains on Docker Swarm while Temporal runs on K8s, dynamic-schdlr needs to reach temporal:7233. Options:

1. NodePort — expose Temporal frontend on a fixed port on K8s nodes. Set TEMPORALIO_HOST to the K8s node IP and TEMPORALIO_PORT to the NodePort
2. LoadBalancer — if available, expose via cloud LB. Set TEMPORALIO_HOST to the LB address
3. Shared network — if K8s nodes are on the same network as Swarm nodes, use a ClusterIP with kube-proxy or a headless service

Upgrade procedure

# Update values if schema changes are needed (createDatabase/manageSchema handle it)
helm upgrade temporal \
  --repo https://go.temporal.io/helm-charts \
  -f temporal-values.yaml \
  --namespace temporal \
  --timeout 900s

The Helm chart runs a schema migration Job automatically as a pre-upgrade hook.

Dynamic config

# In temporal-values.yaml
server:
  dynamicConfig:
    limit.maxIDLength:
      - value: 255
        constraints: {}

Changes applied via helm upgrade or directly via kubectl apply on the ConfigMap (auto-picked up without pod restart).

---

Summary

	Docker Swarm (auto-setup)	Kubernetes (Helm chart)
Readiness	Ready now, minimal changes	Requires K8s cluster availability
Schema management	Automatic on every start	Kubernetes Job on install/upgrade
Scaling	Single replica, all roles	Per-role independent scaling
Official support	Community examples	First-class Temporal-maintained
Monitoring	Manual Prometheus setup	Preconfigured Grafana dashboards
Migration path	Swarm → K8s: same databases, just re-point	N/A

Recommendation: Deploy via Docker Swarm now. Plan K8s migration when the cluster is operational across all environments. The databases are portable — switching from auto-setup to Helm chart just means pointing the Helm values at the same PostgreSQL.