fix: speed up CI image builds with BuildKit layer caching and a shared cross-PR cache

[Wallgau]

Speed up CI image builds with BuildKit layer caching + a shared cross-PR cache

Summary
Reworks the build-images job in test-ci.yml to stop rebuilding unchanged Docker layers and to share a global layer cache across PRs. The expensive images (Langflow's torch/pip layer, backend's uv sync layer) are now restored from cache instead of rebuilt from scratch.

What changed
Replaced whole-image tar caching with BuildKit layer caching. Dropped the get_image_hash.py → actions/cache → docker save/docker load tar shuttle and the custom build_or_load_image/pull_image bash. Builds now go through docker/build-push-action@v6 with cache-from/cache-to, so only the layers that actually changed rebuild. A flows-only change no longer invalidates the dependency-install layer.
Direct push to GHCR. Non-fork builds push straight to ghcr.io//openrag-:${sha} — no local load, no tarball artifact for internal PRs.

Global shared registry cache. Each image keeps a fixed cache tag ghcr.io//openrag-:buildcache (type=registry, not branch-scoped, no 10 GB Actions-cache cap). main writes it; every internal PR reads it — so a PR that doesn't touch an image gets that image's layers for free on its first run.
main seeds the cache cheaply. Added a push: [main] trigger so a merge re-warms :buildcache. The test-suite, e2e-run, e2e-merge-reports, and the tests/e2e gate jobs are gated off push, so a merge to main runs only build-images (cache warming) — not the full 8-shard e2e suite.
Per-PR cache retained. PRs still write a per-PR gha cache, so a second commit in the same PR reuses its own layers even for images it did change.
Fork behavior unchanged. Fork PRs still build locally and upload the image as an artifact (the :buildcache package stays private, so forks build cold — same as today).

Impact
First push to main after merge does one cold build (~7 min for Langflow) to seed :buildcache.
After that, a fresh PR off main that doesn't modify an image restores it straight from the shared cache — Langflow drops from ~7 min to ~1–2 min on the first run, with no per-PR warm-up needed.
Merges to main no longer pay for a full test/e2e run.
Notes / follow-ups
Cache freshness tracks main: if main bumps an image's deps, the next main push rebuilds that layer once (~7 min on main), and subsequent PRs inherit the warm layer.
scripts/ci/get_image_hash.py is no longer referenced by CI and can be removed in a follow-up if unused elsewhere.

Test plan
Open this PR → confirm build-images runs 4 images in parallel and passes.
Confirm test-suite + e2e run on the PR as before.
Merge → confirm the main push runs only build-images (no e2e) and populates the openrag-*:buildcache packages.
Open a follow-up PR off main that doesn't touch Langflow → confirm the Langflow build is a cache hit and finishes in ~1–2 min.

Summary by CodeRabbit

• Chores

  • Updated CI image workflows to build, cache, pull, and publish images more efficiently.
  • CI image builds now run on pushes to the main branch, while heavier test and end-to-end jobs remain excluded.
  • Added retry handling when pulling prebuilt images.

• Style

  • Improved spacing in the environment configuration section.

• Documentation

  • Added an internal TODO comment for CI testing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 80510727-72b3-422c-bad8-7f9f1bc99173

📥 Commits

Reviewing files that changed from the base of the PR and between a874a2e and 0412220.

📒 Files selected for processing (2)

• Makefile
• frontend/components/chat-renderer.tsx

---

Walkthrough

The CI workflow now warms image layers on pushes to main, routes matrix images through explicit BuildKit build or DockerHub pull handling, and skips test and E2E jobs for push events.

Changes

CI image build and distribution

Layer / File(s)	Summary
Event routing and job gating .github/workflows/test-ci.yml	Adds a main push trigger, permits build-images on push events, and excludes push events from test and E2E jobs.
Image build and distribution .github/workflows/test-ci.yml	Determines per-image Dockerfile, target reference, cache settings, and mode; builds with BuildKit or pulls, retries, retags, and optionally publishes DockerHub images.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitHub
  participant BuildImages
  participant BuildKit
  participant DockerHub
  participant GHCR
  GitHub->>BuildImages: trigger on main push or eligible workflow event
  BuildImages->>BuildKit: build selected image
  BuildKit->>GHCR: publish non-fork build
  BuildImages->>DockerHub: pull prebuilt image when mode is pull
  DockerHub-->>BuildImages: return image
  BuildImages->>GHCR: publish retagged image when not a fork

Loading

Possibly related PRs

• langflow-ai/openrag#1962: Modifies the same workflow’s image build and pull logic.
• langflow-ai/openrag#2087: Updates architecture-aware image caching and build flow in the same workflow.
• langflow-ai/openrag#2102: Reworks the same matrix image build, pull, retag, and caching logic.

Suggested labels: refactor

Suggested reviewers: lucaseduoli

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: faster CI image builds using BuildKit layer caching and shared caches.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch build-improvement

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

.github/workflows/test-ci.yml (1)

171-180: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Reuse the shared registry cache for fork PRs.

The fork fallback step currently hardcodes the GitHub Actions cache (type=gha), which means fork PRs will miss the shared registry cache seeded by the main branch.

Consider reusing the computed cache_from parameter so that fork PRs can also hit the registry cache. If the GHCR package is public, BuildKit will fetch unchanged layers anonymously, significantly speeding up the first build for external contributors. If it is private, BuildKit safely ignores the cache miss with a non-fatal warning and falls back to type=gha.

⚡ Proposed refactor

       - name: Build image (fork fallback)
         if: steps.params.outputs.mode == 'build' && env.IS_FORK == 'true'
         uses: docker/build-push-action@v6
         with:
           context: .
           file: ${{ steps.params.outputs.dockerfile }}
           load: true
           tags: ${{ steps.params.outputs.image }}
-          cache-from: type=gha,scope=${{ matrix.image }}
+          cache-from: ${{ steps.params.outputs.cache_from }}
           provenance: false

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/test-ci.yml around lines 171 - 180, Update the “Build
image (fork fallback)” step to use the computed cache_from parameter for
cache-from instead of hardcoding type=gha, while preserving the existing fork
fallback conditions and build settings.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/test-ci.yml:
- Around line 171-180: Update the “Build image (fork fallback)” step to use the
computed cache_from parameter for cache-from instead of hardcoding type=gha,
while preserving the existing fork fallback conditions and build settings.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 654a32bf-5ea9-474b-b9ed-361d52d8ad7d

📥 Commits

Reviewing files that changed from the base of the PR and between 4fc2c56 and a874a2e.

📒 Files selected for processing (1)

• .github/workflows/test-ci.yml

[github-actions]

React Doctor found no new issues. 🎉

_{Reviewed by React Doctor for commit 0412220.}

[lucaseduoli]

Nice work!!