LLM-survey-political-compass

Questionnaire from politicalcompass.org

This project runs questionnaire prompts against one or more LLMs, writes the answers and scores to JSONL, and gives you input that can be plotted afterward.

The repository includes two example questionnaires:

• political-compass-org
• simple-two-question-example

Install

bun install

Run The CLI

Show the CLI help:

bun src/index.ts --help

List registered questionnaires:

bun src/index.ts --list-questionnaires

List bundled models:

bun src/index.ts --list-models

Run one questionnaire against one model:

bun src/index.ts \
  --questionnaire simple-two-question-example \
  --model openai/gpt-5.3-chat \
  --run-id demo-run

Available models include openai/gpt-5.3-chat and anthropic/claude-sonnet-4.6. More available at Vercel AI Gateway Models

Vercel AI Gateway & Costs

• API key: Create a .env file at the project root and set AI_GATEWAY_API_KEY to your Vercel AI SDK (AI Gateway) key. Example:

AI_GATEWAY_API_KEY=your_vercel_ai_sdk_key_here

• Free credit: Vercel currently provides a $5 free credit for new AI SDK keys. One full questionnaire run costs roughly $0.13 USD, so the free credit covers multiple runs.

• Rate limits: The free $5 credit is rate-limited. If you hit rate limits or see throttling, increase the request delay by adjusting BASE_DELAY in src/consts.ts. Raising BASE_DELAY will slow requests and help stay within the free-tier rate limits.

• Safety tip: Monitor usage and costs when running large batches or multiple models.

Run multiple questionnaires and models with comma-separated values:

bun src/index.ts \
  --questionnaire political-compass-org,simple-two-question-example \
  --model openai/gpt-5.3-chat,anthropic/claude-sonnet-4.6 \
  --run-id multi-run

Output

Results are written inside the questionnaire folder as JSONL files:

src/questionnaires/<questionnaire-directory>/results-<run-id>.jsonl

Each run produces records with these types:

• answer for each model response
• score for the final questionnaire result

Each question chooses its own answer strings from question.answers[].text. The CLI uses those values when it asks the model and when it validates the response.

The score record uses the shape returned by the questionnaire scorer. Political compass returns economic and social. Other questionnaires can return a different numeric shape, such as total.

Adding your own questionnaire

This adds a simple two-question example with custom answer strings and a custom scoring function that sums up points fields on the answer objects.

1. Create The Data Types

File: src/questionnaires/simple-two-question-example/types.ts

import type { QuestionnaireData } from "../types";

export interface SimpleTwoQuestionExampleData extends QuestionnaireData {}

This example does not require extra questionnaire metadata. If you want custom fields, add them to this interface.

2. Create The Scorer

File: src/questionnaires/simple-two-question-example/scorer.ts

import type { Score } from "../../models/Questionnaire";
import type { QuestionnaireScoreContext, QuestionnaireScorer } from "../types";
import type { SimpleTwoQuestionExampleData } from "./types";

export function calculateSimpleTwoQuestionExampleScore({
 answers,
}: QuestionnaireScoreContext<SimpleTwoQuestionExampleData>): Score {
 let total = 0;

 for (const { answer } of answers) {
  total += Number(answer.points ?? 0);
 }

 return {
  total,
 };
}

export const simpleTwoQuestionExampleScorer: QuestionnaireScorer<SimpleTwoQuestionExampleData> = {
 calculateScore: calculateSimpleTwoQuestionExampleScore,
};

The scorer reads custom answer data from answer.points and returns a single numeric key named total.

3. Create The Questionnaire Definition

File: src/questionnaires/simple-two-question-example/index.ts

import { createQuestionnaireDefinition } from "../createQuestionnaireDefinition";
import { simpleTwoQuestionExampleScorer } from "./scorer";
import type { SimpleTwoQuestionExampleData } from "./types";

const simpleTwoQuestionExampleData: SimpleTwoQuestionExampleData = {
 id: "simple-two-question-example",
 title: "Simple Two Question Example",
 preprompt: "Respond only with valid JSON.",
 questions: [
  {
   question: "Pick a drink.",
   answers: [
    { idx: 0, text: "tea", points: 1 },
    { idx: 1, text: "coffee", points: 2 },
   ],
  },
  {
   question: "Pick a pet.",
   answers: [
    { idx: 0, text: "cat", points: 3 },
    { idx: 1, text: "dog", points: 4 },
   ],
  },
 ],
};

export const simpleTwoQuestionExampleDefinition = createQuestionnaireDefinition({
 data: simpleTwoQuestionExampleData,
 scorer: simpleTwoQuestionExampleScorer,
 directoryName: "simple-two-question-example",
});

This is the questionnaire data itself. The answer strings are tea, coffee, cat, and dog, so the model is not constrained to the political compass answer set.

4. Register The Questionnaire

File: src/questionnaires/index.ts

import { politicalCompassOrgDefinition } from "./politicalcompass.org/index";
import { simpleTwoQuestionExampleDefinition } from "./simple-two-question-example/index";

export const questionnaireDefinitions = {
 [politicalCompassOrgDefinition.id]: politicalCompassOrgDefinition,
 [simpleTwoQuestionExampleDefinition.id]: simpleTwoQuestionExampleDefinition,
} satisfies Record<string, AnyQuestionnaireDefinition>;

5. Run The Example

bun src/index.ts \
  --questionnaire simple-two-question-example \
  --model openai/gpt-5.3-chat \
  --run-id simple-example-run

This writes output to:

src/questionnaires/simple-two-question-example/results-simple-example-run.jsonl

The score record for this questionnaire looks like this:

{ "score": { "total": 5 } }

6. Add A Test

File: src/questionnaires/simple-two-question-example/scorer.test.ts

import { describe, expect, test } from "bun:test";
import type { QuestionnaireResponse } from "../../models/Questionnaire";
import { calculateSimpleTwoQuestionExampleScore } from "./scorer";
import type { SimpleTwoQuestionExampleData } from "./types";

describe("calculateSimpleTwoQuestionExampleScore", () => {
 test("sums custom answer points", () => {
  const data: SimpleTwoQuestionExampleData = {
   id: "simple-two-question-example",
   title: "Simple Two Question Example",
   preprompt: "",
   questions: [],
  };

  const responses: QuestionnaireResponse[] = [
   {
    question: { question: "Question 1", answers: [] },
    answer: {
     idx: 0,
     text: "tea",
     points: 1,
    },
   },
   {
    question: { question: "Question 2", answers: [] },
    answer: {
     idx: 1,
     text: "dog",
     points: 4,
    },
   },
  ];

  expect(
   calculateSimpleTwoQuestionExampleScore({
    answers: responses,
    data,
   }),
  ).toEqual({
   total: 5,
  });
 });
});

Political Compass Example

The political compass questionnaire is registered as political-compass-org.

Files

• Questionnaire data: src/questionnaires/politicalcompass.org/questions.json
• Questionnaire registration: src/questionnaires/politicalcompass.org/index.ts
• Scorer: src/questionnaires/politicalcompass.org/scorer.ts
• Types: src/questionnaires/politicalcompass.org/types.ts

Run It

bun src/index.ts \
  --questionnaire political-compass-org \
  --model openai/gpt-5.3-chat \
  --run-id political-compass-demo

Result Shape

The scorer returns:

{
 economic: number,
 social: number,
}

The questionnaire data uses its own answer strings, and the scorer can read extra fields on each answer entry, such as points.

Adding Your Own Questionnaire

To add a new questionnaire:

1. Create a folder under src/questionnaires/.
2. Define questionnaire data with id, title, preprompt, and questions.
3. Put the answer strings you want in question.answers[].text.
4. Add any custom JSON fields you want to use in scoring.
5. Create a scorer module that implements QuestionnaireScorer.
6. Register the questionnaire in src/questionnaires/index.ts.
7. Add a scorer test.
8. Run it with bun src/index.ts --questionnaire <id> --model <model> --run-id <run-id>.

Testing

Run the type checker:

bun run typecheck

Run the full test suite:

bun test

Run the focused tests for the CLI and scorers:

bun test src/cli.test.ts src/questionnaires/politicalcompass.org/scorer.test.ts src/questionnaires/simple-two-question-example/scorer.test.ts

Project Structure

• src/index.ts - CLI entrypoint
• src/cli.ts - argument parsing and help text
• src/runner.ts - questionnaire/model execution and result writing
• src/questionnaires/index.ts - questionnaire registry
• src/questionnaires/createQuestionnaireDefinition.ts - helper for defining questionnaires
• src/questionnaires/politicalcompass.org/ - political compass example
• src/questionnaires/simple-two-question-example/ - two-question example with custom answer strings and custom fields
• generate_plots.ipynb - plotting notebook