Qualitative content evaluation toolkit - Discover, measure, and calibrate quality dimensions in text content.
Most evaluation systems start with a rubric and score content against it. This toolkit inverts that:
content → discover quality dimensions → calibrate measurement → evaluate → feedback
You can enter at any point. You don't need a predefined rubric. The system can discover what dimensions of quality exist in your content and help you build measurement tools around them.
What dimensions of quality exist in this content?
- Blind Summary - Assess quality without any rubric, identifying strengths/weaknesses organically
- Theme Extraction - Find recurring quality themes across a corpus of content
How do we measure those dimensions?
- Rubric Definition - Formal criteria with rating levels and point ranges
- Pairwise Comparison - Direct A vs B quality judgments ("which is better?")
- Bradley-Terry Ranking - Derive ordinal rankings from pairwise comparisons
Is our measurement accurate?
- Prediction Delta - Compare blind assessment to rubric scores (detects leniency)
- Ceiling Detection - Find score compression at top of scale
- Coverage Gaps - What quality dimensions is the rubric missing?
Apply measurement to content
- Score - Apply rubric criteria to content
- Rank - Produce ordinal rankings via pairwise comparisons
Generate useful output
- Quantitative - Scores, distributions, statistics
- Qualitative - Strengths, growth areas, recommendations
Evaluate content quality without defining criteria upfront. The blind summary approach identifies what matters in the content itself.
Analyze "themes not captured by rubric" across a corpus to discover what your rubric should include but doesn't.
Find when your rubric can't distinguish quality differences. Bradley-Terry comparisons revealed 580x quality differences among submissions that all scored 9/10 on a rubric.
Even when scores are maxed out, generate specific feedback about why top performers are top, and where they can still grow.
Run evaluation multiple times and aggregate results with confidence weighting. Flag high-variance items for human review.
pip install qual-evalfrom qual_eval.discovery import BlindSummary
from qual_eval.measurement import pairwise_compare, bradley_terry_rank
from qual_eval.calibration import detect_ceiling_effects
# Discover quality dimensions without a rubric
summary = BlindSummary(content="...")
print(summary.strengths)
print(summary.weaknesses)
print(summary.overall_quality)
# Compare two pieces of content directly
winner = pairwise_compare(content_a, content_b, criterion="clarity")
# Rank a corpus using Bradley-Terry
rankings = bradley_terry_rank(contents, criterion="overall_quality")
# Check if your rubric has ceiling effects
ceiling_report = detect_ceiling_effects(scores, rubric)- Education: Grade assignments, discover what rubrics miss, generate student feedback
- Content Moderation: Evaluate content quality at scale without rigid rules
- Hiring: Assess written work samples with calibrated criteria
- Research: Analyze qualitative data with reproducible measurement
- Product: Evaluate user-generated content, reviews, documentation
qual_eval/
├── discovery/ # What dimensions of quality exist?
│ ├── blind_summary.py
│ └── theme_extraction.py
├── measurement/ # How do we measure them?
│ ├── rubric.py
│ ├── pairwise.py
│ └── bradley_terry.py
├── calibration/ # Is measurement accurate?
│ ├── prediction_delta.py
│ ├── ceiling_detection.py
│ └── coverage_gaps.py
├── evaluation/ # Run the eval
│ ├── score.py
│ └── rank.py
└── feedback/ # Generate output
├── quantitative.py
└── qualitative.py
MIT