feat(glooko): granular SSV2 cursor sync (experimental, off by default)#429
Draft
ryceg wants to merge 16 commits into
Draft
feat(glooko): granular SSV2 cursor sync (experimental, off by default)#429ryceg wants to merge 16 commits into
ryceg wants to merge 16 commits into
Conversation
Adds an opt-in UseSsv2Sync mode to the Glooko connector that sources data
from the mobile app's granular SSV2 endpoints (cursor-paginated, per-record)
instead of the web graph/batch flow.
- Cursor store: sync_cursors jsonb column on connector_configurations
(migration AddConnectorSyncCursors) + IConnectorSyncCursorStore contract
and tenant-scoped implementation. Persists {lastUpdatedAt, lastGuid} per
tenant/connector/resource for incremental resume across syncs/restarts.
request.To != null => explicit-range backfill (bypass cursor); To == null
=> incremental (resume + persist).
- Generic FetchSsv2Async<TPage,TRecord> cursor pagination.
- Glucose from cgm/egvs (new TransformEgvsToSensorGlucose).
- Boluses / scheduled+temporary+suspend basals / carbs / manual BG / foods
from a cursor-fed GlookoBatchData through the existing mappers
(FetchAndMapViaV2Async split into a shared, behavior-preserving
MapAndPublishV2BatchAsync).
- Device events from pumps/events (new GlookoPumpEventMapper; unrecognised
event kinds are skipped, not guessed).
- Off by default; existing v2/v3 windowed path unchanged. 69 unit tests pass.
Deferred: pumps/alarms, pumps/settings, extended/injection boluses (need live
field-shape capture), tombstone deletes (no publisher delete surface yet).
…e, accurate docs Addresses the review findings on the SSV2 cursor path before it can become a default: - Capture the final page's cursor. The loop broke on LastPage *before* advancing the cursor, so a completed sync persisted the second-to-last watermark and re-fetched the last page every run. Advance from each page (incl. the last) before the stop check, and only persist when the cursor actually moved from where the run started (no more writing a no-op epoch cursor). Covered by new pagination + loop-guard tests. - Per-resource isolation. Each SSV2 resource fetch is now wrapped so one failing feed (network/5xx/decode) is logged and skipped instead of aborting the whole pass — mirroring the windowed batch path's contract. Previously a single resource error lost already-fetched glucose too. - egvs no longer special-cases startDate. Confirmed live that Glooko's egvs endpoint ignores startDate when a cursor is present (cursor is authoritative) and works without it, so egvs now follows the same incremental-vs-backfill rule as every other resource. - Corrected the UseSsv2Sync doc: it claimed "glucose-only" but the path syncs every data type and bypasses the v2/v3 windowed path (and UseV3Api) entirely. - Documented the sendSoftDeleted=false stance (source deletions are not propagated yet) as a deliberate, tracked decision, and noted the CLI path doesn't persist cursors. Tests: SSV2 pagination/loop-guard/per-resource-isolation (GlookoSsv2SyncTests) and the cursor store round-trip/multi-resource/missing-config/corrupt-json (ConnectorSyncCursorStoreTests).
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| private sealed class FixedGlookoTokenProvider : GlookoAuthTokenProvider | ||
| { | ||
| public FixedGlookoTokenProvider() | ||
| : base(new HttpClient(), new ConnectorTokenCache(), |
Comment on lines
+33
to
+66
| foreach (var evt in events) | ||
| { | ||
| if (evt.SoftDeleted) continue; | ||
|
|
||
| var eventType = MapEventType(evt.Type); | ||
| if (eventType == null) | ||
| { | ||
| skipped++; | ||
| continue; | ||
| } | ||
|
|
||
| var date = ParseTimestamp(evt.PumpTimestamp); | ||
| if (date == null) continue; | ||
|
|
||
| // Stable across timezone re-correction: prefer Glooko's guid, else the raw fake-UTC string. | ||
| var key = !string.IsNullOrEmpty(evt.Guid) | ||
| ? $"glooko_event_{evt.Guid}" | ||
| : $"glooko_event_raw_{evt.Type}_{evt.PumpTimestamp}"; | ||
|
|
||
| var now = DateTime.UtcNow; | ||
| results.Add(new DeviceEvent | ||
| { | ||
| Id = Guid.CreateVersion7(), | ||
| Timestamp = date.Value, | ||
| LegacyId = key, | ||
| SyncIdentifier = key, | ||
| Device = _connectorSource, | ||
| DataSource = _connectorSource, | ||
| EventType = eventType.Value, | ||
| Notes = evt.Type, | ||
| CreatedAt = now, | ||
| ModifiedAt = now | ||
| }); | ||
| } |
Comment on lines
+52
to
+82
| foreach (var egv in egvs) | ||
| { | ||
| if (egv.Calculated || egv.SoftDeleted || egv.GlucoseValue <= 0) continue; | ||
|
|
||
| var date = ParseV2Timestamp(egv.DisplayTime); | ||
| if (date == null) continue; | ||
|
|
||
| // mg/dL × 100, same integer encoding as the v2 cgm/readings feed. | ||
| var mgdl = egv.GlucoseValue / 100.0; | ||
|
|
||
| // Prefer Glooko's guid; else the RAW fake-UTC display string (stable across timezone | ||
| // re-correction, unlike corrected ticks) so a re-sync upserts rather than duplicates. | ||
| var key = !string.IsNullOrEmpty(egv.Guid) | ||
| ? $"glooko_egv_{egv.Guid}" | ||
| : $"glooko_egv_raw_{egv.DisplayTime}"; | ||
|
|
||
| var now = DateTime.UtcNow; | ||
| results.Add(new SensorGlucose | ||
| { | ||
| Id = Guid.CreateVersion7(), | ||
| Timestamp = date.Value, | ||
| LegacyId = key, | ||
| SyncIdentifier = key, | ||
| Device = _connectorSource, | ||
| DataSource = _connectorSource, | ||
| Mgdl = mgdl, | ||
| Direction = ParseTrendToDirection(egv.TrendArrow), | ||
| CreatedAt = now, | ||
| ModifiedAt = now | ||
| }); | ||
| } |
Mines two more SSV2 endpoints toward retiring the heavy v3 graph path: pumps/injection_boluses → Bolus and pumps/injection_basals → BasalInjection. These are manual pen injections — the SSV2 counterpart to the v3 graph's gkInsulinBolus/gkInsulinBasal series, and the *only* insulin source for MDI (pen) users, who previously got nothing from the SSV2 path. Closes the biggest gap blocking SSV2 from replacing v3. - New GlookoInjectionInsulin model + page wrappers. Confirmed via the decompiled app that injection_boluses/basals extend the same GKPumpObject envelope as normal_boluses (pumpTimestamp/guid/softDeleted/updatedAt — verified live), plus insulinDelivered + name (insulin product). - GlookoV4TreatmentMapper.MapSsv2InjectionInsulin mirrors MapV3ManualInsulin: resolves the insulin name against InsulinCatalog for DIA/peak (long-acting for basal, rapid-acting for bolus), keys LegacyId/SyncIdentifier on the stable Glooko guid (raw-timestamp hash fallback), skips soft-deleted and non-positive units. - Fetched per-resource via the isolation-wrapped path, gated on Boluses/BasalInjections. Both feeds were empty on the available (pump) test account, so the model is derived from the decompiled GKInjectionEvent + the verified shared envelope; a wire mismatch would yield no records (safe), not corruption. Needs an MDI account to live-confirm. Tests: GlookoV4TreatmentMapperSsv2InjectionTests (units/name/context, guid-keyed id, hash fallback, soft-delete/zero skip, timestamp correction). Glooko suite: 78 green.
Collaborator
Author
Update: mined two more SSV2 endpoints — pen insulinAdded Verified via the decompiled app that SSV2 coverage now: egvs, normal_boluses, scheduled/temp/suspend basals, meter readings, foods, pumps/events, injection boluses + basals. Remaining v3 gaps to mine (toward retiring v3):
|
Two more SSV2 endpoints mined toward retiring the v3 graph path: - pumps/alarms → SystemEvent (GlookoSystemEventMapper.TransformSsv2AlarmsToSystemEvents). SSV2 counterpart to the v3 pumpAlarm series. Unlike the camelCase pump feeds these are snake_case Mongo docs (verified live): `value` is the alarm code (e.g. "raw_occlusion"), `alarm_severity` the severity — mapped straight to SystemEventType (hazard/warning/info, else Alarm) rather than the v3 path's keyword heuristic. Published under the DeviceEvents count alongside pumps/events, matching the v3 path. - cgm/carbs_events → CarbIntake (MapSsv2CarbsEvents). SSV2 counterpart to the v3 carbAll series — standalone app-logged carbs, additional to bolus.carbsInput + foods (additive ItemsSynced). carbs_events was empty on the test account, so modeled from the decompiled GKCgmCarbsEvent (`cgmCarbs` + `eventTime`/`timestamp`); a wire mismatch yields no records. Both keyed on the stable Glooko guid (raw-timestamp fallback), soft-delete aware, fetched through the per-resource isolation wrapper. SSV2 coverage now closes the gkInsulin*, carbAll and pumpAlarm v3 gaps; remaining for full v3 parity: extended_boluses, normal_boluses type→Automatic, and pump-mode state spans. Tests: GlookoSystemEventMapperSsv2AlarmTests + GlookoV4TreatmentMapperSsv2CarbsTests. Glooko suite: 90 green.
Collaborator
Author
Update: mined
|
Mines pumps/extended_boluses → Bolus (MapSsv2ExtendedBoluses): an immediate portion (initialDelivery) plus a portion delivered over a duration (extendedDelivery over extendedBolusDuration). Both present → Dual, otherwise Square. Net-new — neither the windowed path nor the v3 graph has an extended-bolus series. - New GlookoExtendedBolus model (shares the GKBolus/GKPumpObject envelope + extended fields). - Total insulin from insulinDelivered, falling back to initial+extended; guid-keyed LegacyId/SyncIdentifier; soft-delete aware; fetched via the isolation wrapper. - extendedBolusDuration → Bolus.Duration (minutes). Feed is empty on the test account, so the duration UNIT is an assumption (alt: seconds) flagged in the model — confirm against a pump that delivers extended boluses. A wrong unit only mis-scales duration, not other data. - Also tidied the carbs_events publish to use the additive PublishRecordTypeAsync (consistency + proper publish-failure health handling). Carbs on an extended meal bolus are not decomposed here yet (follow-up). Tests: GlookoV4TreatmentMapperSsv2ExtendedBolusTests (square/dual, sum fallback, null duration, soft-delete/zero skip). Glooko suite: 95 green.
Pump operating-mode spans (auto/manual/sleep/exercise/boost/...) are the one piece with NO SSV2 source. Verified by reverse-engineering the Glooko app: the only mode data it fetches is the AGGREGATE `graph/statistics/overall` (per-mode percentages + duration strings — no timestamps, no spans); the app never calls graph/data; and no SSV2 resource (pumps/readings|events|settings, stats-from-server) carries a mode field. Per-interval mode spans exist only as the v3 graph/data pump*Mode series. So the SSV2 path keeps ONE slim v3 graph/data call requesting ONLY the ~26 pump*Mode series (confirmed ~14KB/week, returns real spans with type/duration/start/end), fed into the existing, tested TransformV3PumpModeToStateSpans mapper. Windowed to the last few days on incremental syncs (modes don't change retroactively; dedup absorbs overlap), full range on backfill. Wrapped so a failure (incl. a stale-code 403) degrades to no spans rather than failing the sync. Net: SSV2 now covers every Glooko record stream; v3 graph/data is reduced from ~40 series to a mode-only call (~95% lighter) but cannot be eliminated entirely — pump-mode spans have no other source. Mapper already covered by GlookoStateSpanMapperPumpModeTests; mode-only fetch live-verified against a CamAPS account. Glooko suite: 95 green.
Collaborator
Author
Update: extended boluses + pump-mode state spans (commits 1d36e05, ad41b32)
v3-parity status
Conclusion: SSV2 covers every Glooko record stream; v3 Remaining (optional / net-new): daily_insulin_summaries, hypo_events, app-logged cgm/insulin_events, Glooko unit suite: 95 green. |
Map the SSV2 pumps/settings feed (basal/bolus programs) to a Nocturne Profile so the SSV2 sync path no longer needs the v3 devices_and_settings call for profiles. Segment times are seconds-of-day; basal rate U/hr and carb ratio g/U pass through; ISF/target-BG are mg/dL x 100 (scaled down); DIA is seconds (10800 -> 3h). Uses the most-current, non-soft-deleted snapshot with a guid-stable id. The v3 device-settings Profiles block in PerformSyncInternalAsync is now guarded with !config.UseSsv2Sync; the SSV2 path fetches+publishes profiles inside FetchAndMapViaSsv2Async. Non-SSV2 (v2/v3) behaviour is unchanged.
Add GlookoSsv2Device model + pump/cgm page envelopes, the Ssv2PumpsPath/Ssv2CgmDevicesPath constants, and GlookoDeviceMapper mapping both hardware-inventory feeds to V4 PatientDevice records (InsulinPump vs CGM by feed, properties.{pumpModel,cgmModel} preferred for the human model, guid-keyed deterministic UUIDv5 Id for upsert stability, soft-delete aware, lastSyncTimestamp fake-UTC corrected).
Wire the fetch+map into FetchAndMapViaSsv2Async under the DeviceEvents gate (no SyncDataType exists for hardware inventory). The mapped devices are logged but NOT published: IDevicePublisher exposes only device-status and device-event methods, with no PatientDevice upsert path. The mapper + model + tests are ready for whenever that publisher method lands.
…rcises, exercise_events) Add SSV2 sync support for the app-logged feeds used by CGM-only/MDI Glooko users: - cgm/insulin_events (snake_case) -> Bolus/BasalInjection via GlookoV4TreatmentMapper.MapSsv2InsulinEvents. fast_acting -> rapid Bolus; long_acting/intermediate/basal -> long-acting BasalInjection; unknown -> Bolus. Uses display_time (falls back to event_time); DIA/peak resolved by category. - notes (camelCase) -> Note via new GlookoNoteMapper. - exercises (camelCase, duration seconds) -> Activity via new GlookoActivityMapper. - cgm/exercise_events (snake_case, duration minutes, string intensity) -> Activity. Exercise durations normalized to minutes (what Activity.Duration expects): exercises /60, exercise_events unchanged. Intensity kept as a string (numeric source stringified). All records guid-keyed (raw-timestamp hash fallback), soft-delete aware, per-resource isolation via FetchSsv2SafelyAsync. Wired into FetchAndMapViaSsv2Async under Boluses/BasalInjections, Notes, and Activity activeTypes gates; added path constants and InitializeMappers fields.
# Conflicts: # src/Connectors/Nocturne.Connectors.Glooko/Configurations/GlookoConstants.cs # src/Connectors/Nocturne.Connectors.Glooko/Models/GlookoSsv2Models.cs # src/Connectors/Nocturne.Connectors.Glooko/Services/GlookoConnectorService.cs
# Conflicts: # src/Connectors/Nocturne.Connectors.Glooko/Configurations/GlookoConstants.cs # src/Connectors/Nocturne.Connectors.Glooko/Models/GlookoSsv2Models.cs
- carbs_events: the wire field is "carbs" (not "cgmCarbs") and timestamps are displayTime/eventTime (no "timestamp") — confirmed live on an app-logging account. The previous model never deserialized the carb value (always 0 → all dropped). Fixed the JsonPropertyName + added displayTime (preferred), so standalone carbs actually import. - PatientDevices: added IDevicePublisher.PublishPatientDevicesAsync + DevicePublisher impl (upsert-by-deterministic-Id via IPatientDeviceRepository) and wired the connector to publish the mapped pumps/cgm_devices instead of only logging them. Glooko suite 140 green; API builds.
Comment on lines
+86
to
+96
| foreach (var basal in settings.BasalSettings) | ||
| { | ||
| if (basal.Segments is not { Length: > 0 }) | ||
| continue; | ||
|
|
||
| var profileData = GetOrCreateProfileData(store, ResolveName(basal.ProfileName, basal.ProfileId), dia); | ||
| profileData.Basal = basal.Segments | ||
| .OrderBy(s => s.Start) | ||
| .Select(s => SecondsToTimeValue(s.Start, s.Rate)) | ||
| .ToList(); | ||
| } |
Comment on lines
+68
to
+72
| catch (Exception ex) | ||
| { | ||
| _logger.LogError(ex, "Failed to publish PatientDevice records for {Source}", source); | ||
| return false; | ||
| } |
Comment on lines
+52
to
+55
| catch (Exception e) | ||
| { | ||
| _logger.LogWarning(e, "[{ConnectorSource}] Error mapping SSV2 exercise", _connectorSource); | ||
| } |
Adds three SSV2 health/biometric feeds to the Glooko connector, gated under
SyncDataType.Activity (no dedicated weight/steps/HR SyncDataType exists):
- Body weight (BOTH sources → BodyWeight, normalized to kg):
- /api/v2/weights (manual/HealthKit): "value" is GRAMS (86700 → 86.7 kg).
- /api/v2/validic/weights (third-party): "weight" already KILOGRAMS.
- Step count → StepCount: /api/v2/validic/routines daily "steps" (rounded;
Source bit 0 set = absolute total, not a delta).
- Resting heart rate → HeartRate: /api/v2/validic/biometric_measurements
"restingHeartrate". This is the ONLY heart-rate field Glooko exposes anywhere
(confirmed by decompiling: GKValidicBiometricMeasurement.restingHeartrate +
the serversync ModelUtil resource registry; there is NO continuous/time-series
HR stream). Panel records without it are skipped.
Models + *Page envelopes (casing per feed), path constants, and three mappers
(GlookoBodyWeightMapper/StepCountMapper/HeartRateMapper) with guid-keyed
deterministic GUID ids, soft-delete + non-positive skipping, and timeMapper
correction.
Publisher plumbing (mirrors the PatientDevice commit): added
IMetadataPublisher.Publish{BodyWeights,StepCounts,HeartRates}Async + impls in
MetadataPublisher (per-record upsert by deterministic Id via the existing
IBodyWeightService/IStepCountService/IHeartRateService get-by-id+create/update),
plus Publish{BodyWeight,StepCount,HeartRate}DataAsync helpers in
BaseConnectorService, called from the connector. The deterministic id is a real
GUID carried in model.Id because the entity OriginalId column only accepts
24-char Mongo ObjectIds, so a GUID is the upsert key that round-trips to the PK.
Glooko suite 155 green (15 new); Glooko + API build clean.
Collaborator
Author
Update: large SSV2 expansion (parallel agent batch) — commits up to cbb3784Mined the rest of the high-value SSV2 catalog (built across isolated worktrees, then merged + integrated): Treatments / app-logged: Profiles (2nd v3 call retired): Devices: Biometrics: Bug fixed: Shapes were captured live from several tenant accounts (most feeds are empty on a pump-only account; insulin_events/exercises/weights/etc. needed app-logging accounts to confirm wire shape + casing — which is inconsistent per endpoint). Glooko unit suite: 155 green; connector + API build clean. A full read-only review of the whole expansion is running now; I'll post its findings. Still behind the off-by-default |
Comment on lines
+226
to
+230
| catch (Exception ex) | ||
| { | ||
| _logger.LogError(ex, "Failed to publish BodyWeight records for {Source}", source); | ||
| return false; | ||
| } |
Comment on lines
+262
to
+266
| catch (Exception ex) | ||
| { | ||
| _logger.LogError(ex, "Failed to publish StepCount records for {Source}", source); | ||
| return false; | ||
| } |
Comment on lines
+298
to
+302
| catch (Exception ex) | ||
| { | ||
| _logger.LogError(ex, "Failed to publish HeartRate records for {Source}", source); | ||
| return false; | ||
| } |
…timestamp order Review follow-ups: - MUST-FIX: Notes, Activity and the health feeds (body weight / step count / heart rate, gated under Activity) were unreachable dead code — SyncDataType.Notes/Activity were absent from Glooko's SupportedDataTypes, so they could never enter activeTypes. Added Notes + Activity (and reconciled ManualBG/TempBasals) to both SupportedDataTypes lists (the service property and the [ConnectorRegistration] attribute). Added orchestration regression tests asserting a Notes/Activity request actually fetches those feeds. - carbs_events timestamp: GetRawGlookoDate prefers its 2nd arg, so the field order was wrong (DisplayTime could be overridden by the legacy timestamp). Reordered to prefer DisplayTime, matching the insulin_events/exercise_events mappers. Verified (not bugs): DIA — SSV2 active_insulin_time is seconds (÷3600→hours), v3's is already hours, both yield 3.0h. ISF/target ÷100 confirmed against live settings (9008→90, 10809→108 mg/dL). Health publishers omit SystemAuditScope by design — consistent with MetadataPublisher (systemEvents also omit it); attribution comes from the background service's DbContext.AuditContext. Glooko suite 157 green.
Collaborator
Author
Review complete + must-fixes applied (commit 863990b)A full read-only review of the expansion ran. Outcome: Fixed:
Verified, not bugs:
Still open before flipping
Glooko unit suite: 157 green. |
Comment on lines
+92
to
+95
| catch (Exception e) | ||
| { | ||
| _logger.LogWarning(e, "[{ConnectorSource}] Error mapping SSV2 exercise event", _connectorSource); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Glooko's granular SSV2 sync path — per-resource
/api/v2/{resource}endpoints with persisted(lastUpdatedAt, lastGuid)cursors — behindUseSsv2Sync(default false). Draft: gated experimental, not for default until the checklist below is cleared.Why SSV2
cgm/egvs,pumps/events).patient=param, so it's structurally immune to theglookoCode-change 403 (data_cant_view) class fixed in fix(glooko): re-authenticate when a changed glookoCode triggers 403 data_cant_view #428.Hardening in this branch (review follow-ups)
LastPagefirst, persisting the second-to-last watermark and re-fetching the last page every run. Now persists only when the cursor actually moved from the run's start (no no-op epoch writes).startDate— live-confirmed Glooko ignoresstartDatewhen a cursor is present (cursor authoritative) and egvs works without it; egvs now follows the same incremental/backfill rule as the other resources, no special case.UseSsv2Syncsummary (it syncs all types and bypasses the v2/v3 path; not "glucose-only"); documented thesendSoftDeleted=falsestance and that the CLI path doesn't persist cursors.Tests
GlookoSsv2SyncTests— multi-page pagination + final-cursor persistence, non-advancing-cursor loop guard, per-resource isolation.ConnectorSyncCursorStoreTests— round-trip, multi-resource coexistence, unknown resource, corrupt JSON, missing-config no-op.Pre-default checklist (remaining)
sendSoftDeleted=false); decide whether to ingest tombstones + soft-delete downstream before flipping the default.main(reconcile with the fix(glooko): re-authenticate when a changed glookoCode triggers 403 data_cant_view #428 re-auth changes toGlookoConnectorService).AddConnectorSyncCursorsreviewed (additive nullablejsonb, metadata-only — safe on 17.x).