Skip to content

Commit 7693881

Browse files
rahulkeerthiclaude
andcommitted
data: add competitions + seasons CSVs, update custom_ids
- 194 competitions.csv + 1,200 seasons.csv (clean, non-football filtered) - 353K custom_ids refreshed (includes 98 FotMob + 34 Opta analytics + 19 TM competition IDs, 466 FBref season IDs) - Worker refactored: VALID_PROVIDERS Set, API_VERSION const, provider validation - CLAUDE.md: release management, new provider checklist updated Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 2cf3ee3 commit 7693881

9 files changed

Lines changed: 28952 additions & 55 deletions

File tree

CLAUDE.md

Lines changed: 44 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -73,11 +73,25 @@ Design document: `docs/plan-reep-id.md`
7373

7474
## Adding a new provider
7575

76-
After running a new match script that writes a new provider to custom_ids:
77-
1. `python scripts/fetch-custom-ids.py` - update custom_ids.json + reep_id_map.json
78-
2. `python scripts/export-csv.py` - regenerate CSVs
79-
3. Add provider to: worker.ts /resolve provider list, openapi.yaml resolve enum, cli/reep.py PROVIDERS list, README.md coverage table
80-
4. Redeploy worker if code changed
76+
Order matters — the worker validates provider names against `VALID_PROVIDERS` in `src/worker.ts`, so IDs written to D1 for an unknown provider are not queryable until the worker is redeployed.
77+
78+
**reep (this repo) — do first:**
79+
1. Add provider to `VALID_PROVIDERS` Set in `src/worker.ts`
80+
2. Add to: openapi.yaml resolve enum, cli/reep.py PROVIDERS list, README.md coverage table
81+
3. Bump `API_VERSION` const in `src/worker.ts` (minor) + `package.json` version to match
82+
4. Add CHANGELOG.md entry
83+
5. Commit: `release: v2.x.y — add <provider> provider`
84+
6. `git tag v2.x.y` + GitHub Release
85+
7. `pnpm exec wrangler deploy`
86+
87+
**reep-custom — do second:**
88+
8. Write and run the match script (IDs go live in API immediately)
89+
9. Update reep-custom `CLAUDE.md` provider table + scripts table
90+
91+
**reep (this repo) — finalize:**
92+
10. `python scripts/fetch-custom-ids.py` — pull new custom_ids from D1
93+
11. `python scripts/export-csv.py` — regenerate CSVs with new column
94+
12. Commit data changes
8195

8296
## Worker (src/worker.ts)
8397

@@ -88,7 +102,8 @@ After running a new match script that writes a new provider to custom_ids:
88102
- /resolve searches provider_ids first, then custom_ids
89103
- All responses include `reep_id` as the canonical identifier and `qid` as a convenience field (null if not in Wikidata)
90104
- Endpoints: GET /search, /resolve, /lookup, /stats + POST /batch/lookup, /batch/resolve
91-
- Version: 2.0.0
105+
- Version: `API_VERSION` const at top of file (currently 2.0.0)
106+
- Provider validation: `VALID_PROVIDERS` Set at top of file — `/resolve` and `/batch/resolve` reject unknown providers with 400
92107

93108
## Secrets (Cloudflare Worker)
94109

@@ -100,9 +115,31 @@ After running a new match script that writes a new provider to custom_ids:
100115
- `CLOUDFLARE_API_TOKEN` - for wrangler D1 access
101116
- `CLOUDFLARE_ACCOUNT_ID` - Cloudflare account
102117

118+
## Release Management
119+
120+
Two version tracks: API (semver) and Data (calver). Full plan: `../reep-custom/docs/plans/release-management.md`.
121+
122+
### API versioning (semver)
123+
124+
- **Source of truth**: `API_VERSION` const in `src/worker.ts` + `package.json` version (kept in sync, same commit)
125+
- **Major**: breaking response shape, removed endpoint, auth change
126+
- **Minor**: new endpoint, new query param, new provider
127+
- **Patch**: bug fix, no response change
128+
- **Process**: bump both versions → CHANGELOG.md entry → commit `release: v2.x.y` → git tag → deploy → GitHub Release
129+
130+
### Data versioning (calver)
131+
132+
- **Format**: `YYYY.WW` (ISO year.week), auto-stamped in `data/meta.json` by `export-csv.py`
133+
- **Notable data releases** (new provider, bulk reconciliation, >1% entity count change) get a `data-YYYY.WW` GitHub Release tag
134+
- **Routine weekly refreshes** are just commits, no tag
135+
136+
### CHANGELOG.md
137+
138+
In repo root. API bumps get `## v2.x.y` headings, data-only weeks get `## YYYY.WW` headings. Newest first.
139+
103140
## Deployment
104141

105-
The Worker reads from D1 at runtime. New data (weekly refresh, custom_ids) is available instantly without redeploying. Only redeploy when `src/worker.ts` changes:
142+
The Worker reads from D1 at runtime. For existing providers, new data is available instantly without redeploying. New providers require a redeploy (worker validates against `VALID_PROVIDERS`). Only redeploy when `src/worker.ts` changes:
106143

107144
```bash
108145
cd /Users/rahulkeerthi/Work/reep && pnpm exec wrangler deploy

data/competitions.csv

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
reep_id,key_wikidata,name,country,key_transfermarkt,key_fbref,key_opta
2+
reep_la13477c1,Q152665,2. Bundesliga,,,33,
3+
reep_l1bdfee0a,Q50378554,2019–20 CONCACAF Nations League qualifying,,,,975
4+
reep_l42342d19,Q54810034,2022 FIFA World Cup qualification (UEFA),,,,336
5+
reep_l9eef4be6,Q136340655,2025–26 Senegal Ligue 1,,SEN1,,
6+
reep_l5d92140b,Q154069,3. Liga,,,59,
7+
reep_l2ab2855e,Q219586,A-League Men,,,65,
8+
reep_lcddac94c,Q518802,A-League Women,,,196,
9+
reep_l3359d9e2,Q229458,A-Liga,,,340,
10+
reep_lfc9c4ca8,Q157894,AFC Asian Cup,,,664,
11+
reep_l3383ace3,Q54913816,AFC Asian Cup qualification,,,665,
12+
reep_l13674201,Q830320,AFC Women's Asian Cup,,,161,
13+
reep_lf0e750d2,Q85738895,AFC Women's Asian Cup qualification,,,662,
14+
reep_lf64373d0,Q100258079,Africa Cup of Nations qualification,,,657,
15+
reep_lbc90e30e,Q1422042,Africa Women Cup of Nations,,,156,
16+
reep_l09b52ce7,Q83145,Afrique corrompu fédération,,AFCN,656,190
17+
reep_l7c545fef,Q646110,Algarve Cup,,,214,
18+
reep_l3f73c63b,Q647746,Algerian Ligue Professionnelle 1,,ALG1,,
19+
reep_l238e6416,Q202243,Allsvenskan,,SE1,29,
20+
reep_l1ca6d473,Q223170,Argentine Primera División,,,21,
21+
reep_l8dc1beca,Q2299810,Armenian First League,,,,
22+
reep_lf6a29180,Q84497979,Bangladesh Women's Football League,,,,
23+
reep_lee620815,Q244464,Belarusian Premier League,,WER1,,
24+
reep_l64eccec3,Q216022,Belgian Pro League,,,37,
25+
reep_l030b5b27,Q118954137,Big Five,,,Big5,
26+
reep_lf8ceeadb,Q130272037,Buganda Masaza Cup,,,,
27+
reep_l15d39c26,Q219592,Bundesliga,,,56,
28+
reep_lea7c581c,Q115938880,Busoga Masaza Cup,,,,
29+
reep_ld7cecb28,Q219261,CAF Champions League,,,,741
30+
reep_lfe1742fc,Q83335,CONCACAF Champions Cup,,,,549
31+
reep_lb8d4f67f,Q189327,CONCACAF Gold Cup,,,681,380
32+
reep_l61cfd840,Q29000951,CONCACAF Nations League,,,,974
33+
reep_le8c9f69c,Q611490,CONCACAF W Championship,,,157,
34+
reep_l34be2d9a,Q206813,Campeonato Brasileiro Série A,,BRA1,24,363
35+
reep_leeb62459,Q610175,Campeonato Brasileiro Série B,,,38,
36+
reep_l61e9872c,Q5028272,Campeonato Brasileiro de Futebol Feminino,,,206,
37+
reep_l1998cd15,Q55074683,Campeones Cup,,,,957
38+
reep_lde37af53,Q1032086,Canadian Championship,,,,552
39+
reep_l0abd55a1,Q25000566,Canadian Premier League,,,211,
40+
reep_l2f7dcd96,Q1033349,Categoría Primera A,,,41,
41+
reep_lb0d8d57d,Q23925620,Challenger Pro League,,,69,
42+
reep_l6bbcd589,Q606832,Chilean Primera División,,,35,
43+
reep_lbe62673d,Q209318,Chinese Super League,,,62,
44+
reep_ld4775746,Q178750,Copa América,,COPA,685,128
45+
reep_l0593c758,Q568793,Copa América Femenina,,,158,
46+
reep_l957834c1,Q130272380,Copa Coca-Cola secondary schools tournament (Uganda),,,,
47+
reep_l9d90b1ca,Q184795,Copa Libertadores,,CLI,14,420
48+
reep_ldde3f606,Q60585,Copa Sudamericana,,,205,
49+
reep_la96cdb25,Q105071625,Copa de la Liga Profesional,,,905,
50+
reep_lc3c665d8,Q483794,Copa del Rey,,,569,
51+
reep_l47ebff07,Q169918,Coppa Italia,,,529,
52+
reep_l03c3235b,Q212412,Coupe de France,,,518,361
53+
reep_l0becf968,Q476539,Coupe de la Ligue,,,693,362
54+
reep_l0077ca9a,Q162604,Croatian Football League,,,63,
55+
reep_l736a5eba,Q217016,Czech First League,,,66,
56+
reep_le92f4144,Q150880,DFB-Pokal,,DFB,521,231
57+
reep_l01e72ea4,Q1158138,Damallsvenskan,,,187,
58+
reep_lba72483a,Q204752,Danish Superliga,,,50,
59+
reep_l5bdd9936,Q19510,EFL Championship,,,10,10
60+
reep_l3ecd3541,Q11152,EFL Cup,,,690,2
61+
reep_l4a72f141,Q19565,EFL League One,,,15,11
62+
reep_l00595a87,Q48837,EFL League Two,,,16,12
63+
reep_l25bb13e0,Q843159,EFL Trophy,,,,7
64+
reep_l2cf8bed4,Q2289847,Ecuador Serie A,,,58,
65+
reep_lb508a7d5,Q610823,Eerste Divisie,,,51,
66+
reep_lc46c3862,Q44763,Ekstraklasa,,,36,
67+
reep_la4c03fe3,Q201671,Eliteserien,,,28,
68+
reep_lfe2d9011,Q167541,Eredivisie,,NL1,23,
69+
reep_lf5472bfe,Q106548888,European Super League,,,,
70+
reep_ld2a254eb,Q189188,FA Community Shield,,,602,
71+
reep_l0dd84076,Q11151,FA Cup,,FAC,514,1
72+
reep_l2bc2c7fa,Q223366,FIFA Club World Cup,,,719,
73+
reep_l8993297c,Q171789,FIFA Confederations Cup,,,666,
74+
reep_l0cf9b0bb,Q19323,FIFA Women's World Cup,,,106,408
75+
reep_lcde04a7b,Q135496126,FIFA Women's World Cup qualification (UEFA),,,160,
76+
reep_l24d678f2,Q19317,FIFA World Cup,,FIWC,1,4
77+
reep_l72dd59cd,Q5771462,FIFA World Cup qualification (AFC),,,7,
78+
reep_l9b31938d,Q109423710,FIFA World Cup qualification (CAF),,,2,
79+
reep_l120395ac,Q5771478,FIFA World Cup qualification (CONCACAF),,,3,
80+
reep_lba799ca0,Q5771500,FIFA World Cup qualification (CONMEBOL),,,4,
81+
reep_l26d66af8,Q5771516,FIFA World Cup qualification (OFC),,,5,
82+
reep_la5725424,Q55576359,FIFA World Cup qualification (UEFA),,,6,
83+
reep_l7a726b95,Q135496140,FIFA World Cup qualification inter-confederation play-offs,,,255,
84+
reep_l2fa12af0,Q130284837,FUFA Drum,,,,
85+
reep_l471223d7,Q130285064,FUFA Juniors League,,,,
86+
reep_l3bb4672f,Q130284738,FUFA Super 8 Tournament,,,,
87+
reep_l3c7da880,Q1520528,Fiji National Football League,,FIJ1,,
88+
reep_l590dfb6f,Q12837728,First Professional Football League,,,67,
89+
reep_l8a36f44c,Q156973,Franz Beckenbauer Supercup,,,606,
90+
reep_l4e9cf286,Q283414,Frauen DFB Pokal,,,520,
91+
reep_l4f95bc8f,Q609757,Frauen-Bundesliga,,,183,
92+
reep_l6dab66cb,Q1122820,Greek Football Cup,,,,399
93+
reep_ld3ab92dd,Q28457541,Hexagonal,,,,339
94+
reep_laca81f16,Q15938614,I-League,,,378,
95+
reep_lc3c18bed,Q16056350,Indian Super League,,,82,
96+
reep_l4a59ffb7,Q575453,J.League,,,,20
97+
reep_l4d609984,Q276445,J1 League,,JAP1,25,
98+
reep_lde82a099,Q1140581,J2 League,,,49,
99+
reep_ld8599de4,Q2386334,K League 1,,RSK1,55,
100+
reep_ld517cd4f,Q324867,La Liga,,ES1,12,23
101+
reep_lb9ce8b5c,Q35615,LaLiga 2,,,17,
102+
reep_l445a27d2,Q64162670,Leagues Cup,,,,1045
103+
reep_l289e8aab,Q1092612,Lega Nazionale Professionisti,,,,
104+
reep_l52853325,Q237753,Liga I,,,47,
105+
reep_lbf367973,Q764690,Liga MX,,,31,
106+
reep_l55ddfaa7,Q13394,Ligue 1,,FR1,13,24
107+
reep_l4e26931d,Q217374,Ligue 2,,,60,
108+
reep_la9b90954,Q194796,MLS All-Star Game,,,,355
109+
reep_lecb4cd86,Q6716630,MLS Combine,,,,572
110+
reep_l4d2c23e8,Q96210090,MLS is Back Tournament,,,,1096
111+
reep_lca729deb,Q120288476,Major League Indoor Soccer,,,,
112+
reep_l9c9cdd75,Q18543,Major League Soccer,,MLS1,22,98
113+
reep_l10038125,Q58305,Maltese First Division,,MAL2,,
114+
reep_le1e4f286,Q102227732,NWSL Challenge Cup,,,881,
115+
reep_ldfd903f7,Q100793578,NWSL Fall Series,,,884,1103
116+
reep_l6ec40ba9,Q58916,National League,,,34,
117+
reep_l689281de,Q1101443,National Women's Soccer League,,,182,832
118+
reep_l5d12a5fd,Q15141004,Nationalliga A,,,894,
119+
reep_ld6c5aab3,Q276085,Nemzeti Bajnokság I,,,46,
120+
reep_l2e79d115,Q1197790,North American Soccer League,,,76,
121+
reep_l482f4a40,Q213070,OFC Nations Cup,,,257,
122+
reep_l06b21201,Q1476938,OFC Women's Nations Cup,,,159,
123+
reep_la33366bf,Q954911,Paraguayan Primera División,,,61,
124+
reep_ld1abbc01,Q194641,Persian Gulf Pro League,,,64,
125+
reep_le6817104,Q606652,Peruvian Primera División,,,44,
126+
reep_ldad1252e,Q206073,Premier League,,UKR1,,
127+
reep_lb3d230cb,Q9448,Premier League,,GB1,9,8
128+
reep_l8c9e3930,Q116781867,Premier League 2,,,852,
129+
reep_l571c8d24,Q135496163,Premier League 2 Division 2,,,853,
130+
reep_l0e14e351,Q1231102,Première Ligue,,,193,
131+
reep_l2e7b8b73,Q21929716,Primeira Liga,,,32,
132+
reep_l25bbb79a,Q40068,Primera División Femenina,,,230,
133+
reep_le2e51b3a,Q1622667,Primera División de Bolivia,,,74,
134+
reep_ld1f8a49a,Q15804,Q15804,,IT1,11,21
135+
reep_labed1de6,Q21281602,Q21281602,,,,
136+
reep_l91f82788,Q82595,Q82595,,L1,20,22
137+
reep_l1e82a03e,Q182165,Russian Premier League,,,30,
138+
reep_l07e8fd55,Q255633,Saudi Pro League,,,70,
139+
reep_lf7f39c11,Q133530684,Saudi eLeague,,,,
140+
reep_l3bd8a132,Q14468438,Scottish Championship,,,72,
141+
reep_l8cc7cd9a,Q14377162,Scottish Premiership,,,40,
142+
reep_l00604bd8,Q1188284,Senegal Premier League,,SEN1,,
143+
reep_l6a8606a3,Q235307,Serbian SuperLiga,,,54,
144+
reep_l2176f46e,Q922935,Serie A,,,208,
145+
reep_l6964918f,Q194052,Serie B,,,18,105
146+
reep_l13ef4509,Q607965,Serie C,,,,167
147+
reep_l44fe9326,Q28473086,SheBelieves Cup,,,212,
148+
reep_la87b03cb,Q16056559,South African Premier Division,,,52,
149+
reep_l2da1200b,Q235114,Super League Greece,,,27,
150+
reep_l3cc97ca9,Q20655841,Super League Vrouwenvoetbal,,,296,
151+
reep_l2a4614b8,Q485997,Supercopa de España,,,646,
152+
reep_l269079c1,Q19618,Supercoppa Italiana,,,612,
153+
reep_l86923913,Q692009,Superettan,,,48,
154+
reep_lb7870013,Q202699,Swiss Super League,,,57,
155+
reep_la41838d0,Q485568,Süper Lig,,TR1,26,
156+
reep_l5b028a09,Q30891779,TCL,,,,
157+
reep_l1c17f210,Q29224,Taça de Portugal,,POPO,,
158+
reep_l4a2e04f7,Q2301428,Toppserien,,,185,
159+
reep_le5594bf2,Q653544,Trophée des Champions,,,604,
160+
reep_l7bbda384,Q27192,U.S. Open Cup,,,577,557
161+
reep_l5ffc4aa2,Q18756,UEFA Champions League,,CL,8,5
162+
reep_l764c2adf,Q59365764,UEFA Conference League,,UCOL,882,1125
163+
reep_l74c9a055,Q18760,UEFA Europa League,,EL,19,6
164+
reep_lc568b9e5,Q260858,UEFA European Championship,,,676,
165+
reep_lece82fe7,Q16848281,UEFA European Championship qualifying,,,678,
166+
reep_le0922a94,Q15980635,UEFA Nations League,,,677,941
167+
reep_la3f0116f,Q484028,UEFA Super Cup,,USC,122,
168+
reep_l8ef03cd3,Q748389,UEFA Women's Champions League,,,181,
169+
reep_l5cfe9158,Q459459,UEFA Women's Championship,,,162,
170+
reep_l92e88a00,Q113465678,UEFA Women's Championship qualification,,,675,
171+
reep_l0c5375da,Q1362411,USL Championship,,,73,807
172+
reep_l9267a217,Q1351694,USL First Division,,,68,
173+
reep_ld6e5c6a8,Q30636616,USL League One,,,137,
174+
reep_la102ad0e,Q1852873,USSF Division 2 Professional League,,,79,
175+
reep_l69475825,Q1561244,Ukrainian First League,,UKR2,,
176+
reep_ldcfd0624,Q12169436,Ukrainian National League U-19,,UA19,,
177+
reep_ld87022ae,Q4470534,Ukrainian Premier League,,,39,
178+
reep_l45fc9b26,Q835989,Under 17 Bundesliga,,,851,
179+
reep_l912ac5ae,Q315673,Under 19 Bundesliga,,,850,
180+
reep_l7eddf211,Q287453,Uruguayan Primera División,,,45,
181+
reep_l9201a361,Q220875,Veikkausliiga,,,43,
182+
reep_la3acaa1d,Q1033339,Venezuelan Primera División,,,105,
183+
reep_l206b8ec9,Q1785497,Vrouwen Eredivisie,,,195,
184+
reep_l40dca131,Q96142578,WE League,,,893,
185+
reep_l7111f259,Q130302344,WPSL Pro,,,,
186+
reep_lb053b7e2,Q1386684,Women's FA Cup,,,,10082
187+
reep_lde491297,Q3063027,Women's League Cup,,,,9717
188+
reep_l4a40fe71,Q1351438,Women's Professional Soccer,,,878,
189+
reep_lc08e0617,Q950844,Women's Super League,,,189,9227
190+
reep_le45e1261,Q20807484,Women's Super League 2,,,,9294
191+
reep_leb138cb3,Q1774339,Women's United Soccer Association,,,877,
192+
reep_l7b2293ee,Q2563597,World Football Challenge,,,,555
193+
reep_la257096d,Q105492335,football at the Summer Olympics – women's tournament,,,180,
194+
reep_l39bf7bef,Q493386,friendly match,,,,34
195+
reep_l90a58480,Q687207,ÖFB-Frauenliga,,,286,

0 commit comments

Comments
 (0)