Statistical evaluation of readability across texts generated by large language models and their classical human-written counterparts.
Companion analysis to Story-generation-process. Part of a Master's research project at Whitireia New Zealand (2025).
As LLM-generated content grows, matching output to a target audience — in this case, children — depends on measurable text properties. Readability is the most studied such property. This repository quantifies the gap between human-written fairy tales and their LLM-generated rewrites across 19 models and 6 established readability indices, using classical statistical tools.
Do LLMs reproduce the readability of classical children's fairy tales, and if not, how do they differ across readability dimensions?
- overalldata.xlsx — raw per-metric scores for every (model × tale) pair, plus the original texts
- combined_model_ranking_all.xlsx — aggregated ranking across metrics
Corpus: 16 classical English fairy tales from Project Gutenberg (Henny Penny, Jack Hannaford, The Rose Tree, The Three Sillies, and 12 others).
Sample: each of 19 LLMs rewrote all 16 tales → 304 generated texts + 16 originals.
| Step | Tool | Purpose |
|---|---|---|
| Metric computation | textstat |
6 readability indices per text |
| Distribution check | scipy, matplotlib |
Q-Q plots, normality diagnostics |
| Group comparison | statsmodels |
One-way ANOVA per metric |
| Pairwise comparison | statsmodels (Tukey HSD) |
Which models differ significantly, on which metric |
| Aggregate ranking | pandas |
Synthesising per-metric results into overall performance |
Readability indices analysed:
- Automated Readability Index (ARI)
- Coleman–Liau Index
- Flesch–Kincaid Grade Level
- Flesch Reading Ease
- Gunning Fog
- SMOG Index
15_Fairy_tales_Project_Gutenberg_Text_Readability Levels Across Models and Original Text.ipynb
overview of readability values across all models and originals
ANOVA_Automated_Readability_Index.ipynb
ANOVA_Coleman_Liau_Index.ipynb
ANOVA_Flesch_Kincaid_Grade_Level_.ipynb
ANOVA_Flesch_Reading_Ease.ipynb
ANOVA_Gunning_Fog.ipynb
ANOVA_SMOG_Index.ipynb
one-way ANOVA per readability metric
Tukey's_HSD_tests.ipynb
post-hoc pairwise comparisons across models
Q_Q_Plot_readability_metrics.ipynb
distribution-normality diagnostics
Grouped_metrics_based_results.ipynb
aggregated analysis per metric group
Combined_model_ranking_results.ipynb
synthesis: which models are best / worst per metric
overalldata.xlsx
combined_model_ranking_all.xlsx
datasets consumed by the notebooks
No single model replicates authentic readability; individual LLMs excel in specific dimensions.
Human-written tales form a distinct cluster in ANOVA results across most indices, confirming that current LLMs have not fully closed the gap in age-appropriate narrative generation.
Detailed per-metric rankings and significance results are in Combined_model_ranking_results.ipynb.
pip install textstat statsmodels scipy pandas matplotlib plotly openpyxl jupyter
jupyter notebookThe notebooks load the bundled .xlsx files directly, no external download required.
- Corpus of 16 tales is small; bootstrap / resampling could strengthen the statistics
- Dale-Chall score relies on a fixed difficulty list, limited for children's-domain vocabulary
- Readability indices measure surface complexity, not semantic coherence or narrative quality
- Story-generation-process — pipeline that produced the LLM-generated texts analysed here
- Master's research: The comparative analysis of human-written and large language model generated children's tales: Readability and accuracy assessments of LLM-generated texts, Whitireia New Zealand (2025)
No license is specified; for commercial use, please open an issue.
Language: English · Русский