Add service health endpoints and harden gRPC/HTTP client robustness#1398
Merged
fidencio merged 5 commits intoJun 2, 2026
Merged
Conversation
Member
|
cc @MatiasVara |
Xynnn007
reviewed
Jun 2, 2026
fidencio
force-pushed
the
topic/grpc-health-and-robustness
branch
from
3c36050 to
b2f7aaf
Compare
Xynnn007
approved these changes
Jun 2, 2026
Contributor
Not related to this PR but I noticed some time ago that Trustee's default config does not cache the collateral at all. PCS stopped sending |
Member
Author
Is it okay if we land this one first, @mythi? This is basically the set of commits that I needed in order to get the helm chart working / passing Kata Containers CI. |
Contributor
|
I don't mind this goes first, the PRs are not coupled. Just a comment that if you plan to make some changes to |
mkulke
reviewed
Jun 2, 2026
| match tokio::time::timeout(RVPS_HEALTH_CHECK_TIMEOUT, c.query_reference_value(req)).await { | ||
| Ok(Ok(_)) => Ok(c), | ||
| Ok(Err(e)) => { | ||
| warn!("RVPS connection health check failed, reconnecting: {e}"); |
Contributor
There was a problem hiding this comment.
we don't attempt a reconnect here, so I would move the log entry to the place where we perform retries. see also the Err arm
Until now none of the Trustee services exposed a way for an external orchestrator to tell whether they were actually up and serving traffic. This makes it impossible for an environment such as Kubernetes to drive liveness and readiness probes against them, which in turn means a wedged or still-initialising service cannot be detected and restarted. Wire up the standard gRPC Health service (via tonic-health) into both the attestation-service and the rvps gRPC servers so that they report SERVING once they are ready to answer requests, and add a small GET /healthz route to the KBS HTTP API that returns 200 OK while the server is alive. The tonic-health crate is added as a workspace dependency and enabled through the attestation-service grpc-bin feature and the rvps crate. Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com> Assisted-by: Cursor <cursoragent@cursor.com>
The reference-value lookup performed during token generation runs on a dedicated helper thread, and that thread used to spin up its own current-thread tokio runtime to drive the asynchronous RVPS query. The mobc connection pool and the tonic gRPC channels it manages are created on the main, multi-threaded runtime, so executing pool.get() and the channel I/O from an unrelated runtime delivered their futures and wakeups to a reactor that never owned the underlying resources. The practical effect was that pool.get() could block forever and the whole attestation request would hang. Capture the handle of the currently running runtime before the helper thread is spawned and use it to block_on() the query, keeping the pool and its connections inside the runtime context where they were created. Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com> Assisted-by: Cursor <cursoragent@cursor.com>
Connections handed out by the RVPS connection pool are long lived, and in practice they can be torn down underneath us by the peer or by an intermediary without the client noticing. When that happens the next query made over such a connection either hangs or fails, even though a healthy RVPS is sitting on the other end. Make the pooled gRPC client defensive about this. The channel now enables HTTP/2 keep-alive and caps the idle lifetime of a connection so that dead connections are surfaced and dropped, and every connection handed out by the pool is first validated with a short, time-bounded health query (an unknown key, for which RVPS simply returns Ok(None)). If that probe does not succeed quickly the connection is discarded and a fresh one is established instead of risking a stale one. Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com> Assisted-by: Cursor <cursoragent@cursor.com>
The channel KBS uses to talk to the attestation-service had no timeout configured. Verifying TEE evidence can legitimately take a long time, most notably for Intel TDX where DCAP has to fetch collateral over the network, so the channel must tolerate slow responses; but with no upper bound at all a connection that has silently wedged would keep a request blocked indefinitely with no way to recover. Give the channel a generous 300 second timeout, which comfortably covers even the slow DCAP path while ensuring a genuinely stuck request eventually fails and returns an error instead of hanging forever. Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com> Assisted-by: Cursor <cursoragent@cursor.com>
The NVIDIA verifier talks to the NVIDIA Remote Attestation Service over HTTP both to submit evidence for attestation and to fetch the JWKS used to validate the response. Both requests were issued with a default reqwest client that has no timeout, so a slow or unreachable NRAS endpoint could leave the verifier blocked indefinitely with no way out. Build explicit reqwest clients with request timeouts for these calls: 60 seconds for the attestation request and 30 seconds for the JWKS fetch, so a misbehaving endpoint results in a clean error rather than a hang. Signed-off-by: Fabiano Fidêncio <ffidencio@nvidia.com> Assisted-by: Cursor <cursoragent@cursor.com>
fidencio
force-pushed
the
topic/grpc-health-and-robustness
branch
from
b2f7aaf to
76bcab9
Compare
mkulke
approved these changes
Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series makes the Trustee services more observable and more resilient
when run as long-lived deployments behind an orchestrator.
It adds standard health endpoints so liveness/readiness probes can be wired
up, and fixes a handful of robustness issues in the inter-service gRPC and
the outbound HTTP clients that can otherwise cause requests to hang
indefinitely.
These changes are independent of any particular deployment method and are
sent ahead of the Helm chart work so they can be reviewed on their own.
serviceability: add health and readiness endpoints to AS, RVPS and KBS
The attestation-service and rvps gRPC servers now register the standard
gRPC Health service (via
tonic-health) and reportSERVINGonce ready,and KBS gains a lightweight
GET /healthzHTTP route. This lets externalorchestrators detect a wedged or still-initialising service and restart it.
attestation-service: query RVPS from the runtime that owns the pool
The reference-value lookup ran on a helper thread that spun up its own
current-thread runtime, while the mobc pool and its gRPC channels live on
the main multi-threaded runtime. Driving
pool.get()from an unrelatedruntime delivered wakeups to the wrong reactor and could hang the request
forever. The query now runs on the handle of the runtime that owns the pool.
attestation-service: recycle stale RVPS gRPC connections
Long-lived pooled connections to RVPS could be silently torn down, making
later queries hang or fail. The channel now uses HTTP/2 keep-alive and a
capped idle lifetime, and each pooled connection is validated on checkout
with a short, time-bounded health query before use.
kbs: bound the lifetime of requests on the AS gRPC channel
The KBS→AS channel had no timeout. Evidence verification can be slow
(notably Intel TDX DCAP), but with no upper bound a wedged connection would
block a request indefinitely. A generous 300s timeout covers the slow path
while ensuring stuck requests fail cleanly.
verifier: bound NVIDIA NRAS HTTP requests with timeouts
The NVIDIA verifier's attestation POST and JWKS fetch used a default
reqwestclient with no timeout, so an unreachable NRAS could hang theverifier. Explicit timeouts are now set (60s attestation, 30s JWKS).