Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 27 additions & 25 deletions .github/workflows/action-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
# ============================================================================
# First, generate an AI-powered PR description using prpilot-summary.
# This provides context for the multi-model consensus review.
# Requires secrets: GEMINI_API_KEY
# Requires secrets: GROQ_API_KEY
# ============================================================================
generate-description:
name: "Step 1: Generate PR Description"
Expand All @@ -52,10 +52,10 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4

- name: Check if GEMINI_API_KEY exists
- name: Check if LLM API key exists
id: check_secrets
run: |
if [ -z "${{ secrets.GEMINI_API_KEY }}" ]; then
if [ -z "${{ secrets.GROQ_API_KEY }}" ]; then
echo "has_key=false" >> $GITHUB_OUTPUT
else
echo "has_key=true" >> $GITHUB_OUTPUT
Expand All @@ -67,9 +67,9 @@ jobs:
uses: bishalprasad321/prpilot-summary@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
llm_api_key: ${{ secrets.GEMINI_API_KEY }}
llm_provider: gemini
ai_model: gemini-2.5-flash
llm_api_key: ${{ secrets.GROQ_API_KEY }}
llm_provider: groq
ai_model: groq-1.5-mini
max_diff_lines: "5000"
enable_incremental_diff_processing: "true"
debug: ${{ inputs.debug_enabled || 'false' }}
Expand All @@ -89,7 +89,7 @@ jobs:
else
echo "⚠️ Description generation encountered an error"
echo "This may be expected if:"
echo " - Secrets are not configured (GEMINI_API_KEY missing)"
echo " - Secrets are not configured (GROQ_API_KEY missing)"
echo " - No meaningful changes in the PR"
echo " - LLM API is temporarily unavailable"
fi
Expand All @@ -100,7 +100,7 @@ jobs:
# ============================================================================
# Run multi-model consensus review after description is generated.
# This calls 3 independent reviewer models + 1 judge model for consensus.
# Requires secrets: GEMINI_API_KEY
# Requires secrets: GROQ_API_KEY
# Outputs: Decision (APPROVE/REQUEST_CHANGES/COMMENT), findings, round count
# ============================================================================
consensus-review:
Expand All @@ -123,31 +123,31 @@ jobs:
PRESET="${{ inputs.model_preset || 'default' }}"
case $PRESET in
high-capability)
echo "reviewer_models=gemini-2.5-pro,gemini-2.5-flash,gemini-2.5-flash-lite" >> $GITHUB_OUTPUT
echo "judge_model=gemini-2.5-pro" >> $GITHUB_OUTPUT
echo "preset_name=High Capability (Best Quality)" >> $GITHUB_OUTPUT
echo "reviewer_models=llama-3.3-70b-versatile,openai/gpt-oss-120b,openai/gpt-oss-20b" >> $GITHUB_OUTPUT
echo "judge_model=openai/gpt-oss-120b" >> $GITHUB_OUTPUT
echo "preset_name=Groq Maximum Quality" >> $GITHUB_OUTPUT
;;
cost-optimized)
echo "reviewer_models=gemini-2.5-flash,gemini-2.5-flash-lite,gemini-2.5-flash-lite" >> $GITHUB_OUTPUT
echo "judge_model=gemini-2.5-flash" >> $GITHUB_OUTPUT
echo "preset_name=Cost Optimized (Lower Cost)" >> $GITHUB_OUTPUT
echo "reviewer_models=llama-3.1-8b-instant,openai/gpt-oss-20b,openai/gpt-oss-20b" >> $GITHUB_OUTPUT
echo "judge_model=openai/gpt-oss-120b" >> $GITHUB_OUTPUT
echo "preset_name=Groq Maximum Speed" >> $GITHUB_OUTPUT
;;
quality-optimized)
echo "reviewer_models=gemini-2.5-pro,gemini-2.5-flash,gemini-2.5-flash-lite" >> $GITHUB_OUTPUT
echo "judge_model=gemini-2.5-pro" >> $GITHUB_OUTPUT
echo "preset_name=Quality Optimized (Best Findings)" >> $GITHUB_OUTPUT
echo "reviewer_models=llama-3.3-70b-versatile,openai/gpt-oss-120b,openai/gpt-oss-20b" >> $GITHUB_OUTPUT
echo "judge_model=openai/gpt-oss-120b" >> $GITHUB_OUTPUT
echo "preset_name=Groq Maximum Quality" >> $GITHUB_OUTPUT
;;
*)
echo "reviewer_models=gemini-2.5-flash,gemini-2.5-flash-lite,gemini-2.5-pro" >> $GITHUB_OUTPUT
echo "judge_model=gemini-2.5-pro" >> $GITHUB_OUTPUT
echo "preset_name=Default (Balanced)" >> $GITHUB_OUTPUT
echo "reviewer_models=llama-3.1-8b-instant,openai/gpt-oss-20b,llama-3.3-70b-versatile" >> $GITHUB_OUTPUT
echo "judge_model=openai/gpt-oss-120b" >> $GITHUB_OUTPUT
echo "preset_name=Groq Default Production" >> $GITHUB_OUTPUT
;;
esac

- name: Check if GEMINI_API_KEY exists
- name: Check if LLM API key exists
id: check_secrets
run: |
if [ -z "${{ secrets.GEMINI_API_KEY }}" ]; then
if [ -z "${{ secrets.GROQ_API_KEY }}" ]; then
echo "has_key=false" >> $GITHUB_OUTPUT
else
echo "has_key=true" >> $GITHUB_OUTPUT
Expand All @@ -159,7 +159,9 @@ jobs:
uses: ./
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
gemini_api_key: ${{ secrets.GEMINI_API_KEY }}
llm_provider: groq
llm_api_key: ${{ secrets.GROQ_API_KEY }}
llm_provider_url: https://api.groq.com/openai/v1
reviewer_models: ${{ steps.model_config.outputs.reviewer_models }}
judge_model: ${{ steps.model_config.outputs.judge_model }}
max_consensus_rounds: "3"
Expand Down Expand Up @@ -198,8 +200,8 @@ jobs:
else
echo "⚠️ Consensus review encountered an error"
echo "This may be expected if:"
echo " - Secrets are not configured (GEMINI_API_KEY missing)"
echo " - Selected models are not valid Gemini v1beta model codes"
echo " - Secrets are not configured (GROQ_API_KEY missing)"
echo " - Selected models are not valid for the current provider"
echo " - API quota exceeded (check API key billing settings)"
echo " - Reviewer/judge models not available"
echo " - LLM API is temporarily unavailable"
Expand Down
129 changes: 72 additions & 57 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# PR Pilot Review

> **Intelligent Multi-Model Consensus AI PR Reviewer** — Automated code reviews with consensus from multiple Gemini AI models.
> **Intelligent Multi-Model Consensus AI PR Reviewer** — Automated code reviews with consensus from configurable LLM providers like Gemini and Groq.

[![Build Status](https://github.com/bishalprasad321/prpilot-review/actions/workflows/ci.yml/badge.svg)](https://github.com/bishalprasad321/prpilot-review/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Overview

PR Pilot Review is a GitHub Action that performs intelligent, consensus-based code reviews on pull requests using Google's Gemini AI models. Unlike single-model reviewers, it runs **3 independent AI reviewers in parallel** and uses a **judge model to reach consensus**, ensuring more thorough and balanced code quality feedback.
PR Pilot Review is a GitHub Action that performs intelligent, consensus-based code reviews on pull requests using configurable LLM providers such as Gemini or Groq. Unlike single-model reviewers, it runs **3 independent AI reviewers in parallel** and uses a **judge model to reach consensus**, ensuring more thorough and balanced code quality feedback.

### Key Features

Expand All @@ -22,20 +22,28 @@ PR Pilot Review is a GitHub Action that performs intelligent, consensus-based co

## Quick Start

### 1. Get a Gemini API Key
### 1. Choose your LLM provider

1. Visit [Google AI Studio](https://makersuite.google.com/app/apikey)
2. Click "Create API Key"
3. Copy the key
This action supports multiple providers today, including `gemini` and `groq`.

### 2. Add to GitHub Secrets
### 2. Add the API key to GitHub Secrets

Go to your repository:

- **Settings** → **Secrets and variables** → **Actions**
- Add secret: `GEMINI_API_KEY` (paste your key)
- Add a provider-specific secret, e.g. `GROQ_API_KEY`

### 3. Use the Action
### 3. Optional base URL for Groq OpenAI-compatible API

If you are using Groq's OpenAI-compatible models, set `llm_provider_url` to:

```yaml
llm_provider_url: https://api.groq.com/openai/v1
```

This will route requests to `https://api.groq.com/openai/v1/chat/completions` for chat-based model inference.

### 4. Use the Action

Create `.github/workflows/ai-review.yml`:

Expand All @@ -61,7 +69,9 @@ jobs:
- uses: bishalprasad321/prpilot-review@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
gemini_api_key: ${{ secrets.GEMINI_API_KEY }}
llm_provider: groq
llm_api_key: ${{ secrets.GROQ_API_KEY }}
llm_provider_url: https://api.groq.com/openai/v1
# Optional: customize model configuration
# reviewer_models: "gemini-2.5-flash,gemini-2.5-flash-lite,gemini-2.5-pro"
# judge_model: "gemini-2.5-pro"
Expand All @@ -79,7 +89,9 @@ All inputs are optional (sensible defaults provided):
with:
# Required
github_token: ${{ secrets.GITHUB_TOKEN }}
gemini_api_key: ${{ secrets.GEMINI_API_KEY }}
llm_provider: groq
llm_api_key: ${{ secrets.GROQ_API_KEY }}
llm_provider_url: https://api.groq.com/openai/v1

# Optional: Model Configuration
reviewer_models: "gemini-2.5-flash,gemini-2.5-flash-lite,gemini-2.5-pro" # 3 models
Expand All @@ -93,47 +105,51 @@ with:
debug: "false" # Verbose logging
```

Use stable Gemini `v1beta` model codes. If you still pass legacy aliases such as `gemini-3.0-flash` or `gemini-3.1-pro`, the action will try to remap them to supported 2.x models before calling the API.
Use stable provider-specific model IDs. For Gemini, use `v1beta` model codes. For Groq, use models like `llama-3.1-8b-instant`, `openai/gpt-oss-20b`, or `llama-3.3-70b-versatile` (check [Groq console](https://console.groq.com/keys) for available models).

### Model Presets

Choose models based on your needs:

#### Default (Recommended) ✅
### Groq Example

```yaml
reviewer_models: "gemini-2.5-flash,gemini-2.5-flash-lite,gemini-2.5-pro"
judge_model: "gemini-2.5-pro"
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
llm_provider: groq
llm_api_key: ${{ secrets.GROQ_API_KEY }}
reviewer_models: "llama-3.1-8b-instant,openai/gpt-oss-20b,llama-3.3-70b-versatile"
judge_model: "openai/gpt-oss-120b"
max_consensus_rounds: "3"
inline_comments_enabled: "true"
```

Best balance of speed, quality, and cost.
### Model Presets

Choose models based on your needs:

#### High Capability (Best Quality)
#### Groq Default Production ✅

```yaml
reviewer_models: "gemini-2.5-pro,gemini-2.5-flash,gemini-2.5-flash-lite"
judge_model: "gemini-2.5-pro"
reviewer_models: "llama-3.1-8b-instant,openai/gpt-oss-20b,llama-3.3-70b-versatile"
judge_model: "openai/gpt-oss-120b"
```

Uses most capable models. Higher cost but best findings.
Best balance of production reliability and coverage.

#### Cost Optimized (Lowest Cost)
#### Groq Maximum Speed ✅

```yaml
reviewer_models: "gemini-2.5-flash,gemini-2.5-flash-lite,gemini-2.5-flash-lite"
judge_model: "gemini-2.5-flash"
reviewer_models: "llama-3.1-8b-instant,openai/gpt-oss-20b,openai/gpt-oss-20b"
judge_model: "openai/gpt-oss-120b"
```

Uses lighter models. Lower cost but lighter analysis.
Optimized for faster review cycles with lighter reviewer models.

#### Quality Optimized (Most Findings)
#### Groq Maximum Quality ✅

```yaml
reviewer_models: "gemini-2.5-pro,gemini-2.5-flash,gemini-2.5-flash-lite"
judge_model: "gemini-2.5-pro"
reviewer_models: "llama-3.3-70b-versatile,openai/gpt-oss-120b,openai/gpt-oss-20b"
judge_model: "openai/gpt-oss-120b"
```

Maximum detection power. Highest cost, most comprehensive reviews.
Highest quality review configuration for the deepest analysis.

## How It Works

Expand Down Expand Up @@ -194,7 +210,7 @@ The action posts a GitHub review with:
See detailed architecture in [docs/adr/](./docs/adr/):

- **ADR-001** — Multi-Model Consensus Architecture
- **ADR-002** — Gemini API Integration & Model Selection
- **ADR-002** — LLM Provider Integration & Model Selection
- **ADR-003** — Inline Comment Mapping
- **ADR-004** — Idempotency & State Management

Expand All @@ -205,7 +221,7 @@ prpilot-review/
├── src/
│ ├── index.ts # Main orchestrator (13-step pipeline)
│ ├── llm/
│ │ └── llm-client.ts # Gemini API abstraction
│ │ └── llm-client.ts # LLM provider abstraction
│ ├── review/
│ │ ├── review-orchestrator.ts # Consensus logic
│ │ └── inline-comment-builder.ts # Line mapping
Expand Down Expand Up @@ -285,12 +301,11 @@ npm run all # Run all checks

## API Keys & Billing

### Getting a Gemini API Key
### Getting an API Key

1. Go to [Google AI Studio](https://makersuite.google.com/app/apikey)
2. Click "Create API Key"
3. Select/create a project
4. Copy the key and add to GitHub repo secrets
1. Choose your provider: `gemini` or `groq`
2. Create an API key in the provider console
3. Copy the key and add it to GitHub repo secrets

### Free Tier Quotas

Expand All @@ -302,48 +317,48 @@ The free tier has limits:
If you hit quota errors:

1. **Upgrade to paid plan** — Recommended for production
2. **Use lighter models** — `gemini-2.5-flash-lite` uses fewer tokens
2. **Use lighter models** — `gemini-2.5-flash-lite` or Groq's smaller models like `llama-3.1-8b-instant` use fewer tokens
3. **Reduce reviewer count** — Use 2 instead of 3 reviewers
4. **Schedule reviews** — Spread runs across off-peak hours

See [quota documentation](https://ai.google.dev/gemini-api/docs/rate-limits).
See provider-specific quota documentation for your chosen LLM.

### Costs

Typical cost per PR review:
Typical cost per PR review will vary by provider and model selection:

- **Default config** — $0.001-0.005 per review (3 reviewers + optional judge)
- **Cost-optimized** — $0.0005-0.002 per review
- **Quality-optimized** — $0.005-0.02 per review
- **Default config** — Low-cost balanced review
- **Cost-optimized** — Minimal token usage and fastest reviewers
- **Quality-optimized** — More comprehensive analysis with higher cost

Pricing based on [Gemini API rates](https://ai.google.dev/pricing).
Pricing is provider-dependent. For Gemini, see [Gemini pricing](https://ai.google.dev/pricing).

## Troubleshooting

### Action not running?

- Check `GEMINI_API_KEY` is set in repo secrets
- Check `llm_provider` and `llm_api_key` are configured correctly
- Verify branch triggers (pull_request, branches)
- Check action logs for errors

### Zero findings even with code changes?

- Enable debug mode to see full model responses
- Check if models are available in v1beta API
- Try different model preset
- Verify PR has actual code changes
- Check whether the selected models are valid for your provider
- Try a different model preset
- Verify the PR has actual code changes

### Quota exceeded errors?

- Check your billing plan at [Google Cloud Console](https://console.cloud.google.com)
- Switch to paid plan or upgrade quota
- Use lighter models (flash-lite variants)
- Check your provider billing or free tier quota
- Switch to paid plan or upgrade quota if available
- Use lighter models where supported

### Model not found errors?

- Verify model names match available list
- Models must be in `v1beta` API
- Check [available models](https://ai.google.dev/gemini-api/docs/models/gemini)
- Verify model names match your provider's available models
- For Gemini, check [Gemini models](https://ai.google.dev/gemini-api/docs/models/gemini)
- For Groq, check your Groq account model docs

### Slow reviews?

Expand Down Expand Up @@ -439,7 +454,7 @@ MIT License — See [LICENSE](./LICENSE) file

## Acknowledgments

- Built with [Gemini AI API](https://ai.google.dev/)
- Built with configurable LLM providers such as Gemini or Groq
- GitHub Actions integration via [@actions/core](https://github.com/actions/toolkit)
- Code bundling via [@vercel/ncc](https://github.com/vercel/ncc)

Expand Down
Loading
Loading