Skip to content

Commit 9e1f391

Browse files
Xynnn007Xynnn007
committed
helm: delete ephemeral admin keys when uninstalling
When use `secrets.useEphemeralGeneratedKeys`, an admin token and asymmetric key pair is generated for auth. Thus commit will automatically delete the auth secrets via a post-delete job. Assisted-by: Cursor <cursoragent@cursor.com> Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
1 parent 9d69ddd commit 9e1f391

5 files changed

Lines changed: 207 additions & 92 deletions

File tree

‎deployment/helm-chart/README.md‎

Lines changed: 23 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ Uninstall the release:
2828
helm uninstall trustee -n coco-trustee
2929
```
3030

31+
> [!NOTE]
32+
> When `secrets.useEphemeralGeneratedKeys` is `true` (default), a **post-delete** Helm hook removes the release-scoped `*-bootstrap-user-keys` Secret automatically.
33+
3134
## Typical scenarios
3235

3336
### Default: LocalFs storage
@@ -68,7 +71,7 @@ When `storageBackend.postgres.mode=external`, the chart does **NOT** deploy the
6871

6972
Key material is controlled only by **`secrets.useEphemeralGeneratedKeys`**:
7073

71-
- **`true` (default):** a Helm **pre-install / pre-upgrade hook** Job generates ephemeral demo keys into a release-scoped Secret (name ends with **`bootstrap-user-keys`**).
74+
- **`true` (default):** a Helm **pre-install / pre-upgrade hook** Job generates ephemeral demo keys into a release-scoped Secret (name ends with **`bootstrap-user-keys`**). **`helm uninstall`** runs a **post-delete** hook that removes that Secret.
7275
- **`false`:** you must **pre-create** a Kubernetes **`Secret`** in the target namespace, then set **`secrets.existingSecretName`** to that name. The bootstrap hook is **not** rendered.
7376

7477
When ephemeral generation is enabled, the hook uses:
@@ -158,26 +161,26 @@ Default **`values.yaml`** is intentionally small. Fixed on-disk paths for **Loca
158161

159162
| Key | Type | Default | Description |
160163
|-----|------|---------|-------------|
161-
| as.affinity | string | `nil` | Affinity and anti-affinity scheduling rules for AS Pods. |
164+
| as.affinity | object | `{}` | Affinity and anti-affinity scheduling rules for AS Pods. |
162165
| as.extraEnvVars | list | `[]` | Extra environment variables for the AS container (for example `HTTP(S)_PROXY` and `NO_PROXY`). |
163166
| as.image.pullPolicy | string | `"Always"` | AS container image pull policy. |
164167
| as.image.repository | string | `"ghcr.io/confidential-containers/staged-images/coco-as-grpc"` | AS container image repository. |
165168
| as.image.tag | string | `"latest"` | AS container image tag. |
166169
| as.imagePullSecrets | list | `[]` | Optional image pull secrets for private registries. |
167-
| as.nodeSelector | string | `nil` | Node label selection constraints for AS Pods. |
168-
| as.podAnnotations | string | `nil` | Extra Pod annotations. |
169-
| as.podSecurityContext | string | `nil` | Pod-level security context overrides. |
170+
| as.nodeSelector | object | `{}` | Node label selection constraints for AS Pods. |
171+
| as.podAnnotations | object | `{}` | Extra Pod annotations. |
172+
| as.podSecurityContext | object | `{}` | Pod-level security context overrides. |
170173
| as.replicaCount | int | `1` | Number of Attestation Service Pod replicas. |
171174
| as.resources | object | `{"limits":{"cpu":"4","memory":"4Gi"},"requests":{"cpu":"500m","memory":"1Gi"}}` | Container CPU/memory requests and limits for AS. |
172-
| as.service.loadBalancerAnnotations | string | `nil` | Annotations applied when `as.service.type` is `LoadBalancer`. |
175+
| as.service.loadBalancerAnnotations | object | `{}` | Annotations applied when `as.service.type` is `LoadBalancer`. |
173176
| as.service.port | int | `50004` | AS Service port. |
174177
| as.service.type | string | `"ClusterIP"` | AS Service type (`ClusterIP` or `LoadBalancer`). |
175178
| as.tolerations | list | `[]` | Tolerations for scheduling AS Pods onto tainted nodes. |
176179
| as.verifier.dcap.collateral_service | string | `"https://api.trustedservices.intel.com/sgx/certification/v4/"` | Intel DCAP collateral service URL. Required when `as.verifier.dcap` is configured. |
177180
| as.verifier.dcap.tcb_update_type | string | `"early"` | DCAP TCB update type (for example `early`). |
178181
| as.verifier.nvidia.type | string | `"Local"` | NVIDIA verifier type: `Local` or `Remote`. When `Remote`, `verifierUrl` must be set. |
179182
| as.verifier.nvidia.verifierUrl | string | `"https://nras.attestation.nvidia.com/v4/attest"` | NRAS URL when `as.verifier.nvidia.type` is `Remote`. |
180-
| bootstrapUserKeysJob | object | `{"keygenImage":{"pullPolicy":"IfNotPresent","repository":"alpine/openssl","tag":"3.5.6"},"kubectlImage":{"pullPolicy":"IfNotPresent","repository":"quay.io/kata-containers/kubectl","tag":"20260112"},"resources":{"limits":{"cpu":"200m","memory":"256Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}}` | Bootstrap hook Job settings (pre-install/pre-upgrade key generation when `secrets.useEphemeralGeneratedKeys=true`). |
183+
| bootstrapUserKeysJob | object | `{"keygenImage":{"pullPolicy":"IfNotPresent","repository":"alpine/openssl","tag":"3.5.6"},"kubectlImage":{"pullPolicy":"IfNotPresent","repository":"quay.io/kata-containers/kubectl","tag":"20260112"},"resources":{"limits":{"cpu":"200m","memory":"256Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}}` | Bootstrap hook Job settings (pre-install/pre-upgrade key generation and post-delete cleanup when `secrets.useEphemeralGeneratedKeys=true`). |
181184
| bootstrapUserKeysJob.keygenImage.pullPolicy | string | `"IfNotPresent"` | OpenSSL `initContainer` image pull policy. |
182185
| bootstrapUserKeysJob.keygenImage.repository | string | `"alpine/openssl"` | OpenSSL `initContainer` image repository that generates demo keys. |
183186
| bootstrapUserKeysJob.keygenImage.tag | string | `"3.5.6"` | OpenSSL `initContainer` image tag. |
@@ -187,13 +190,13 @@ Default **`values.yaml`** is intentionally small. Fixed on-disk paths for **Loca
187190
| bootstrapUserKeysJob.resources | object | `{"limits":{"cpu":"200m","memory":"256Mi"},"requests":{"cpu":"50m","memory":"64Mi"}}` | CPU/memory requests and limits for the bootstrap hook Job. |
188191
| dnsHostAliasWorkaround | bool | `false` | When `true`, templates use Helm `lookup` to write Service `clusterIP` entries into `hostAliases` for clusters that cannot resolve `*.svc.cluster.local`. If Services are missing on first render, rerun `helm upgrade`. |
189192
| fullnameOverride | string | `""` | Override the fully qualified release name (truncated to 63 characters). |
190-
| ingress | object | `{"annotations":null,"className":"","enabled":false,"host":"","tls":[]}` | Optional Kubernetes Ingress for the KBS Service. |
191-
| ingress.annotations | string | `nil` | Ingress annotations. |
193+
| ingress | object | `{"annotations":{},"className":"","enabled":false,"host":"","tls":[]}` | Optional Kubernetes Ingress for the KBS Service. |
194+
| ingress.annotations | object | `{}` | Ingress annotations. |
192195
| ingress.className | string | `""` | IngressClass name. |
193196
| ingress.enabled | bool | `false` | Enable Ingress for KBS. |
194197
| ingress.host | string | `""` | Host-based routing. Leave empty to match all hosts (IP-only access). |
195198
| ingress.tls | list | `[]` | TLS configuration entries. |
196-
| kbs.affinity | string | `nil` | Affinity and anti-affinity scheduling rules for KBS Pods. |
199+
| kbs.affinity | object | `{}` | Affinity and anti-affinity scheduling rules for KBS Pods. |
197200
| kbs.config.admin.audience | string | `"KBS"` | JWT `audience` claim for the bootstrap-generated admin token. |
198201
| kbs.config.admin.issuer | string | `"TrusteeInHelm"` | JWT `issuer` claim for the bootstrap-generated admin token; must match `[admin.authentication.bearer_jwt]`. |
199202
| kbs.config.admin.role | string | `"admin"` | JWT `role` claim and matching `[admin.authorization.regex_acl]` role. |
@@ -206,14 +209,14 @@ Default **`values.yaml`** is intentionally small. Fixed on-disk paths for **Loca
206209
| kbs.image.repository | string | `"ghcr.io/confidential-containers/staged-images/kbs-grpc-as"` | KBS container image repository. |
207210
| kbs.image.tag | string | `"latest"` | KBS container image tag. |
208211
| kbs.imagePullSecrets | list | `[]` | Optional image pull secrets for private registries. |
209-
| kbs.nodeSelector | string | `nil` | Node label selection constraints for KBS Pods. |
210-
| kbs.podAnnotations | string | `nil` | Extra Pod annotations (for example Prometheus scrape or service mesh integration). |
211-
| kbs.podSecurityContext | string | `nil` | Pod-level security context overrides. |
212+
| kbs.nodeSelector | object | `{}` | Node label selection constraints for KBS Pods. |
213+
| kbs.podAnnotations | object | `{}` | Extra Pod annotations (for example Prometheus scrape or service mesh integration). |
214+
| kbs.podSecurityContext | object | `{}` | Pod-level security context overrides. |
212215
| kbs.replicaCount | int | `1` | Number of KBS Pod replicas. |
213216
| kbs.resourceRepository | list | `[]` | KBS resource repository configuration (passed through to KBS config). |
214217
| kbs.resources | object | `{"limits":{"cpu":"2","memory":"2Gi"},"requests":{"cpu":"250m","memory":"256Mi"}}` | Container CPU/memory requests and limits for KBS. |
215218
| kbs.service.exposeLoadBalancer | bool | `false` | When `true`, create an additional external `LoadBalancer` Service (`<fullname>-kbs-lb`). The primary KBS Service (`<fullname>-kbs`) is always internal `ClusterIP`. |
216-
| kbs.service.loadBalancerAnnotations | string | `nil` | Annotations applied to the optional KBS `LoadBalancer` Service when `exposeLoadBalancer=true`. |
219+
| kbs.service.loadBalancerAnnotations | object | `{}` | Annotations applied to the optional KBS `LoadBalancer` Service when `exposeLoadBalancer=true`. |
217220
| kbs.service.port | int | `8080` | Service port for KBS; used by both the internal `ClusterIP` Service and the optional external `LoadBalancer` Service. |
218221
| kbs.tolerations | list | `[]` | Tolerations for scheduling KBS Pods onto tainted nodes. |
219222
| log_level | string | `"info"` | Container `RUST_LOG` for KBS, AS, and RVPS (`info`, `debug`, `warn`, `error`). |
@@ -234,21 +237,21 @@ Default **`values.yaml`** is intentionally small. Fixed on-disk paths for **Loca
234237
| postgresql.primary.persistence.storageClass | string | `""` | StorageClass for the auto-created bundled Postgres PVC; empty uses the cluster default. |
235238
| postgresql.primary.resources | object | `{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"250m","memory":"256Mi"}}` | CPU/memory requests and limits for bundled Postgres. |
236239
| postgresql.service.ports.postgresql | int | `5432` | Bundled Postgres Service port (used in `POSTGRES_URL`). |
237-
| rvps.affinity | string | `nil` | Affinity and anti-affinity scheduling rules for RVPS Pods. |
240+
| rvps.affinity | object | `{}` | Affinity and anti-affinity scheduling rules for RVPS Pods. |
238241
| rvps.extraEnvVars | list | `[]` | Extra environment variables for the RVPS container (for example `HTTP(S)_PROXY` and `NO_PROXY`). |
239242
| rvps.image.pullPolicy | string | `"Always"` | RVPS container image pull policy. |
240243
| rvps.image.repository | string | `"ghcr.io/confidential-containers/staged-images/rvps"` | RVPS container image repository. |
241244
| rvps.image.tag | string | `"latest"` | RVPS container image tag. |
242245
| rvps.imagePullSecrets | list | `[]` | Optional image pull secrets for private registries. |
243-
| rvps.nodeSelector | string | `nil` | Node label selection constraints for RVPS Pods. |
244-
| rvps.podAnnotations | string | `nil` | Extra Pod annotations. |
245-
| rvps.podSecurityContext | string | `nil` | Pod-level security context overrides. |
246+
| rvps.nodeSelector | object | `{}` | Node label selection constraints for RVPS Pods. |
247+
| rvps.podAnnotations | object | `{}` | Extra Pod annotations. |
248+
| rvps.podSecurityContext | object | `{}` | Pod-level security context overrides. |
246249
| rvps.replicaCount | int | `1` | Number of RVPS Pod replicas. |
247250
| rvps.resources | object | `{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"100m","memory":"128Mi"}}` | Container CPU/memory requests and limits for RVPS. |
248-
| rvps.service.loadBalancerAnnotations | string | `nil` | Annotations for the internal RVPS LoadBalancer Service. |
251+
| rvps.service.loadBalancerAnnotations | object | `{}` | Annotations for the internal RVPS LoadBalancer Service. |
249252
| rvps.service.loadBalancerType | string | `"internal"` | Load balancer kind when `rvps.service.type` is `LoadBalancer`: `internal` or `public`. |
250253
| rvps.service.port | int | `50003` | RVPS Service port. |
251-
| rvps.service.publicLoadBalancerAnnotations | string | `nil` | Annotations for the public RVPS LoadBalancer Service when `loadBalancerType=public`. |
254+
| rvps.service.publicLoadBalancerAnnotations | object | `{}` | Annotations for the public RVPS LoadBalancer Service when `loadBalancerType=public`. |
252255
| rvps.service.type | string | `"ClusterIP"` | RVPS Service type (`ClusterIP` or `LoadBalancer`). |
253256
| rvps.tolerations | list | `[]` | Tolerations for scheduling RVPS Pods onto tainted nodes. |
254257
| secrets.existingSecretName | string | `""` | Required when `useEphemeralGeneratedKeys=false`. Secret must contain `KBS_ADMIN_PRIVATE_KEY`, `KBS_ADMIN_PUBKEY`, `AS_TOKEN_SIGNING_PRIVATE_KEY`, and `AS_TOKEN_VERIFICATION_PUBLIC_KEY_CERT_CHAIN`. Optionally include `KBS_ADMIN_TOKEN` (see `kbs/config/docker-compose/setup.sh` for claim layout). |

‎deployment/helm-chart/README.md.gotmpl‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ Uninstall the release:
2828
helm uninstall trustee -n coco-trustee
2929
```
3030

31+
> [!NOTE]
32+
> When `secrets.useEphemeralGeneratedKeys` is `true` (default), a **post-delete** Helm hook removes the release-scoped `*-bootstrap-user-keys` Secret automatically.
3133

3234
## Typical scenarios
3335

@@ -69,7 +71,7 @@ When `storageBackend.postgres.mode=external`, the chart does **NOT** deploy the
6971

7072
Key material is controlled only by **`secrets.useEphemeralGeneratedKeys`**:
7173

72-
- **`true` (default):** a Helm **pre-install / pre-upgrade hook** Job generates ephemeral demo keys into a release-scoped Secret (name ends with **`bootstrap-user-keys`**).
74+
- **`true` (default):** a Helm **pre-install / pre-upgrade hook** Job generates ephemeral demo keys into a release-scoped Secret (name ends with **`bootstrap-user-keys`**). **`helm uninstall`** runs a **post-delete** hook that removes that Secret.
7375
- **`false`:** you must **pre-create** a Kubernetes **`Secret`** in the target namespace, then set **`secrets.existingSecretName`** to that name. The bootstrap hook is **not** rendered.
7476

7577
When ephemeral generation is enabled, the hook uses:

‎deployment/helm-chart/templates/_helpers.tpl‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,9 @@ Each helper truncates `coco-trustee.fullname` to leave room for its suffix befor
8787
{{- define "coco-trustee.names.bootstrapKeys" -}}
8888
{{- printf "%s-bootstrap-keys" (include "coco-trustee.fullname" . | trunc 48 | trimSuffix "-") | trunc 63 | trimSuffix "-" -}}
8989
{{- end }}
90+
{{- define "coco-trustee.names.bootstrapKeysDelete" -}}
91+
{{- printf "%s-bootstrap-keys-del" (include "coco-trustee.fullname" . | trunc 44 | trimSuffix "-") | trunc 63 | trimSuffix "-" -}}
92+
{{- end }}
9093
{{- define "coco-trustee.names.bootstrapUserKeysSecret" -}}
9194
{{- printf "%s-bootstrap-user-keys" (include "coco-trustee.fullname" . | trunc 43 | trimSuffix "-") | trunc 63 | trimSuffix "-" -}}
9295
{{- end }}

‎deployment/helm-chart/templates/bootstrap-user-keys-delete-job.yaml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
{{- if include "coco-trustee.kbs.useBootstrapKeysJob" . | trim }}
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
name: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
6+
labels:
7+
{{- include "coco-trustee.labels" . | nindent 4 }}
8+
annotations:
9+
helm.sh/hook: post-delete
10+
helm.sh/hook-weight: "-20"
11+
helm.sh/hook-delete-policy: hook-succeeded
12+
---
13+
apiVersion: rbac.authorization.k8s.io/v1
14+
kind: Role
15+
metadata:
16+
name: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
17+
labels:
18+
{{- include "coco-trustee.labels" . | nindent 4 }}
19+
annotations:
20+
helm.sh/hook: post-delete
21+
helm.sh/hook-weight: "-18"
22+
helm.sh/hook-delete-policy: hook-succeeded
23+
rules:
24+
- apiGroups: [""]
25+
resources: ["secrets"]
26+
verbs: ["get", "delete"]
27+
---
28+
apiVersion: rbac.authorization.k8s.io/v1
29+
kind: RoleBinding
30+
metadata:
31+
name: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
32+
labels:
33+
{{- include "coco-trustee.labels" . | nindent 4 }}
34+
annotations:
35+
helm.sh/hook: post-delete
36+
helm.sh/hook-weight: "-16"
37+
helm.sh/hook-delete-policy: hook-succeeded
38+
roleRef:
39+
apiGroup: rbac.authorization.k8s.io
40+
kind: Role
41+
name: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
42+
subjects:
43+
- kind: ServiceAccount
44+
name: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
45+
namespace: {{ .Release.Namespace }}
46+
---
47+
apiVersion: batch/v1
48+
kind: Job
49+
metadata:
50+
name: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
51+
labels:
52+
{{- include "coco-trustee.labels" . | nindent 4 }}
53+
annotations:
54+
helm.sh/hook: post-delete
55+
helm.sh/hook-weight: "0"
56+
helm.sh/hook-delete-policy: hook-succeeded
57+
spec:
58+
ttlSecondsAfterFinished: 120
59+
backoffLimit: 2
60+
activeDeadlineSeconds: 120
61+
template:
62+
metadata:
63+
labels:
64+
{{- include "coco-trustee.selectorLabels" . | nindent 8 }}
65+
spec:
66+
restartPolicy: Never
67+
serviceAccountName: {{ include "coco-trustee.names.bootstrapKeysDelete" . }}
68+
automountServiceAccountToken: true
69+
securityContext:
70+
seccompProfile:
71+
type: RuntimeDefault
72+
containers:
73+
- name: delete-user-keys-secret
74+
image: {{ include "coco-trustee.internal.bootstrapKeysKubectlImage" . | quote }}
75+
imagePullPolicy: {{ default "IfNotPresent" (((.Values.bootstrapUserKeysJob | default dict).kubectlImage | default dict).pullPolicy) }}
76+
securityContext:
77+
allowPrivilegeEscalation: false
78+
capabilities:
79+
drop: ["ALL"]
80+
readOnlyRootFilesystem: true
81+
{{- with .Values.bootstrapUserKeysJob.resources }}
82+
resources:
83+
{{- toYaml . | nindent 12 }}
84+
{{- end }}
85+
command: ["/bin/sh", "-c"]
86+
args:
87+
- |
88+
set -eu
89+
NS="$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)"
90+
NAME="{{ include "coco-trustee.userKeysSecretNameResolved" . | trim }}"
91+
kubectl delete secret "$NAME" -n "$NS" --ignore-not-found
92+
echo "Deleted ephemeral user-keys secret $NAME (if present)"
93+
{{- end }}

0 commit comments

Comments
 (0)