feat(reranker): add llamacpp reranker provider for llama.cpp /rerank servers

[chethanuk]

Summary

Adds first-class llamacpp as a reranker provider (HINDSIGHT_API_RERANKER_PROVIDER=llamacpp), so a local llama-server --rerank instance can be used directly instead of spoofing the openrouter provider with a rewritten base URL. Also documents the existing recipe for llama.cpp embeddings via HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai + base URL — the docs-only scope the maintainer confirmed in the issue thread.

Fixes #2665

Root cause

llama.cpp already speaks the Jina/Cohere-compatible /rerank contract that Hindsight's _CohereCompatibleRerankClient implements — the only thing missing was a dispatch branch in create_cross_encoder_from_env() plus config constants. No wire-format work was needed.

How it works

flowchart LR
    subgraph env [Config]
        A["HINDSIGHT_API_RERANKER_PROVIDER=llamacpp<br/>HINDSIGHT_API_RERANKER_LLAMACPP_BASE_URL<br/>(+ optional MODEL / API_KEY / TIMEOUT)"]
    end
    A --> B["create_cross_encoder_from_env()"]
    B -->|"provider == 'llamacpp'"| C["LlamaCppCrossEncoder"]
    B -->|"missing base URL"| E["ValueError at startup<br/>(same shape as tei)"]
    C --> D["_CohereCompatibleRerankClient<br/>POST {base_url}/v1/rerank"]
    D --> F["llama-server --rerank<br/>(reranker GGUF)"]
    F -->|"results[].index remapped<br/>to input order"| C

Loading

Changes

• LlamaCppCrossEncoder composing the existing _CohereCompatibleRerankClient (/rerank contract; permuted results[].index remapping covered by a dedicated test)
• Factory branch with required-base-URL ValueError (same shape as tei)
• Config: ENV_RERANKER_LLAMACPP_* constants, dataclass fields, from_env(), credential fields (base URL + API key)
• Docs: configuration.md + models.mdx reranker sections, plus the llama.cpp embeddings recipe (docs-only, per maintainer scoping)
• .env.example + bundled hindsight-embed template kept in sync (test_bundled_template_matches_repo_root guards this)
• Review fixes folded in: the unsupported-provider error message now lists openrouter (dispatched but missing from the message); skills/hindsight-docs regenerated in-sync with the docs change; score-normalization docstring corrected to match the conditional-sigmoid behavior in reranking.py

How to test

cd hindsight-api-slim
uv run pytest tests/test_llamacpp_cross_encoder.py tests/test_reranker_timeouts.py -v
uv run ruff check hindsight_api/engine/cross_encoder.py hindsight_api/config.py
uv run ty check hindsight_api/engine/cross_encoder.py

# live (optional): llama-server with a reranker GGUF, then
export HINDSIGHT_API_RERANKER_PROVIDER=llamacpp
export HINDSIGHT_API_RERANKER_LLAMACPP_BASE_URL=http://localhost:8080

Test evidence

Run post-rebase on upstream main (ca87e2989), squashed branch:

Check	Result
test_llamacpp_cross_encoder.py + test_reranker_timeouts.py + test_cohere_cross_encoder.py	36 passed
hindsight-embed/tests/test_env_template.py (template sync)	7 passed
ruff check / ruff format --check on touched files	clean
ty check hindsight_api/engine/cross_encoder.py	clean

Notes for reviewers

• Additive only: no schema, API-surface, or default-behavior changes. This provider connects to an existing llama-server --rerank; it does not spawn one (unlike the llamacpp LLM provider).
• Out of scope (follow-ups welcome): a generic RERANK_BASE_URL for arbitrary Jina/Cohere-spec servers (raised by @adoreparler in the issue thread), ollama reranker, per-bank reranker config.

[nicoloboschi]

llamacpp llm provider actually run and manage a llamacpp process so that would not be consistent.

can't you just point use cohere and use custom base url?

[adoreparler]

litellm is also self hosted process and it gets its own variable.