Skip to content

reflector: add generation tracking for stale cache entry detection#921

Open
kaddynator wants to merge 4 commits into
jupyterhub:mainfrom
kaddynator:feat/stale-cache-generation-tracking
Open

reflector: add generation tracking for stale cache entry detection#921
kaddynator wants to merge 4 commits into
jupyterhub:mainfrom
kaddynator:feat/stale-cache-generation-tracking

Conversation

@kaddynator

@kaddynator kaddynator commented Jun 4, 2026

Copy link
Copy Markdown

Background

After a watch reconnect, the reflector cache may still contain entries from the previous watch cycle that have not yet been confirmed by a new event. There is currently no way for callers to distinguish a freshly-confirmed entry from one that may be stale (e.g. a pod that was deleted while the watch was down).

What this adds

A lightweight generation counter on ResourceReflector:

  • self._generation — incremented after each successful _list_and_update, marking a new watch cycle
  • self._entry_generations — a parallel dict stamping each cache entry with the generation in which it was last seen. Raw k8s resource dicts in self.resources are not modified.
  • is_stale(ref_key) — returns True when an entry carries a generation older than the current watch cycle

Thread safety note

self.resources and self._entry_generations are updated on consecutive lines, so there is a brief window of inconsistency between the two dicts. is_stale() is advisory: a caller that races here will either see the old generation (slightly conservative — may re-query the API when it did not need to) or the new one (correct). It will never see an inconsistent state that causes a crash or incorrect action. This matches the existing comment on self.resources that callers accept a potentially out-of-date view.

How to use

pod_key = f"{namespace}/{pod_name}"
if self.pod_reflector.is_stale(pod_key):
    # re-query the API instead of trusting the cached value
    ...

No breaking changes

Existing callers that do not call is_stale() are unaffected. The generation counter adds two small dicts and one integer increment per watch cycle.

kaddynator and others added 4 commits June 3, 2026 20:08
Each watch reconnect increments a _generation counter on the reflector.
Cache entries are stamped with the generation in which they were last
seen via a parallel _entry_generations dict (raw k8s resource dicts are
not modified).

A new is_stale(ref_key) method returns True when an entry's generation
is behind the current watch cycle, indicating it has not been confirmed
since the last reconnect.  Callers such as poll() can use this to decide
whether to re-query the API rather than act on potentially stale data.

No behaviour change for existing callers that do not call is_stale().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant