RAG Evaluation CI/CD System & Analytics Console

This repository implements a production-grade Continuous Integration & Continuous Deployment (CI/CD) Evaluation Pipeline for a Retrieval-Augmented Generation (RAG) system, complete with an Interactive Analytics Dashboard.

The evaluation is based on GateFlow (an AI-powered Smart Campus Access Management System) and uses LLM-as-a-judge metrics (supporting both Grok xAI API and Google Gemini API).

---

🏗️ System Architecture

graph TD
    A[data/document.pdf] -->|Parsed & Chunked| B(RAG Pipeline)
    C[dataset/golden_dataset.json] -->|Test Queries| B
    B -->|Generated Answers & Chunks| D(Evaluation Engine)
    D -->|LLM-as-a-Judge API| E[Grok / Gemini API]
    D -->|Saves Run Results| F[eval_results/run_*.json]
    F -->|Visualized| G(Streamlit Dashboard)
    D -->|Fails Build if below threshold| H[GitHub Actions CI/CD]

Loading

The system comprises the following components:

1. RAG Pipeline (rag/pipeline.py): Extracted from PDF using pypdf, chunked with a sliding window, and retrieved using a custom TF-IDF Vector Search Retriever written from scratch in Python/Numpy.
2. Evaluation Engine (eval/evaluate.py): Executes the test suite against a ground-truth dataset and computes four key metrics on a 1-5 scale using an LLM Judge in a consolidated call.
3. Analytics Dashboard (dashboard/app.py): A Streamlit application built with high-density, dark-themed visual panels, metrics history trends, run comparison (delta checking), and granular question inspectors.
4. CI/CD Workflow (.github/workflows/eval.yml): Automates RAG validation on every code commit and Pull Request, pushing Markdown reports directly to GitHub's Step Summary.

---

📊 LLM-as-a-Judge Evaluation Metrics

We evaluate performance using four core metrics:

• Faithfulness (Groundedness): Assesses whether the generated response is fully derived only from the retrieved context blocks (prevents hallucinations).
• Answer Relevance: Assesses whether the generated response directly, completely, and concisely answers the user's question.
• Context Recall: Assesses whether the retriever successfully fetched all the necessary facts from the source PDF required to answer the question (compares retrieved context against reference answer).
• Semantic Similarity (Correctness): Compares the semantic alignment and facts of the generated answer against the ground-truth reference answer.

---

🚀 Getting Started

1. Prerequisite Setup

Clone the repository and set up a virtual environment:

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

2. Generate Source Document & Golden Dataset

We provide utility scripts to build the environment locally:

# Generate the technical manual PDF (data/document.pdf)
python scripts/generate_sample_pdf.py

# Bootstrap the ground-truth golden dataset (dataset/golden_dataset.json)
python scripts/generate_dataset.py

Tip: To dynamically generate new Q&A pairs from the PDF using the LLM, run python scripts/generate_dataset.py --dynamic (requires API key).

3. Run RAG Evaluation Local Pipeline

Configure your API keys and run the evaluation suite:

# To run using Grok xAI API:
export GROK_API_KEY="your-grok-api-key"
python eval/evaluate.py

# To run using Google Gemini API:
export GEMINI_API_KEY="your-gemini-api-key"
python eval/evaluate.py

# To run offline in Mock Mode (for dry-runs / testing pipelines):
export MOCK_LLM="true"
python eval/evaluate.py

The evaluation script returns exit code 0 if all metric averages exceed the 4.0/5.0 threshold, and exit code 1 if any metric degrades below the target, blocking CI.

4. Launch the Analytics Dashboard

To inspect runs, compare changes, and execute new evaluations interactively:

streamlit run dashboard/app.py

---

🛡️ GitHub Actions CI/CD Integration

The CI/CD workflow is located in .github/workflows/eval.yml. On every push and pull_request, it:

1. Installs the project requirements.
2. Compiles the PDF manual and dataset.
3. Automatically runs the evaluation.
4. Outputs a rich markdown report directly onto the GitHub Actions run summary page.

Setting Repository Secrets

To run live evaluations in CI, navigate to Settings > Secrets and variables > Actions in your GitHub repository and add:

• GROK_API_KEY or XAI_API_KEY (xAI API)
• GEMINI_API_KEY (Google Gemini API)

Note: If no secrets are set, the workflow will automatically default to MOCK_LLM=true so that the build pipeline executes and passes successfully out-of-the-box.