Skip to content

Council members sharing a provider collapse into a single consensus vote #11

Description

@geertvanzoest

Symptom

With several council members configured through the same provider — e.g. three models behind one OpenAI-compatible gateway:

{
  "providers": [
    {"provider": "openrouter", "model": "openai/gpt-5.2", "api_key": "${OPENROUTER_API_KEY}"},
    {"provider": "openrouter", "model": "google/gemini-3.1-pro-preview", "api_key": "${OPENROUTER_API_KEY}"},
    {"provider": "openrouter", "model": "x-ai/grok-4.3", "api_key": "${OPENROUTER_API_KEY}"}
  ],
  "consensus_threshold": 2
}

a review run produces:

  • providers_used: ["openrouter", "openrouter", "openrouter"]
  • quality_ratings: a single "openrouter" key — members overwrite each other
  • individual_issues: one shared "openrouter" bucket
  • consensus_issues: unreachableclassify() tracks voters as a set of provider names (consensus.py: providers.add(review.provider)), so all three members count as one voter while num_providers = len(reviews) = 3; all_agree can never be true
  • agreement between two same-provider members is misclassified as an individual issue

Root cause

Member identity throughout the pipeline is the provider name: ProviderResponse.providerProviderReview.provider → the consensus grouping set and the aggregation dict keys.

Related: resolve_api_keys rebuilds ProviderConfig field by field, silently dropping any field it doesn't enumerate — a hazard for any field later added to ProviderConfig.

Proposed fix

An optional per-entry label that becomes the member's identity in council output, falling back to the provider name (fully backwards compatible). PR follows.

Verified end-to-end against OpenRouter with the three-member config above: with labels, providers_used carries the labels and quality_ratings/individual_issues get three distinct keys, with consensus/majority classification behaving as documented.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions