fix: incremental-update.py — align changed-QIDs discovery with expanded filters

The previous commit aligned build_scoped_ids_query (the second-stage
per-QID fetch) with the shared Phase 1 query shape. But the first-stage
discovery query in fetch_changed_qids was still using the OLD narrow
filters, so even if a newly-reachable entity got edited on Wikidata,
the incremental wouldn't see it.

Two filters expanded to match fetch-wikidata-entities.py:

1. Team discovery — three-path UNION matching build_team_ids_query:
   - subclass of Q476028 (association football club) — existing
   - subclass of Q103229495 (men's association football team) — new
   - P641=Q2736 AND subclass of Q847017 (sports club with football) — new

   Matters because Wikidata classifies many well-known clubs solely
   under Q103229495 (Q7156 Barcelona, Q170703 Boca Juniors) or under
   generic Q847017 (Q8206935 Estudiantes de Río Cuarto). Those were
   silently unreachable by the old filter.

2. Competition discovery — now mirrors the full-fetch competition
   query's shape:
   - Class path requires explicit P641 = Q2736 (blocks NHL/PGA/rugby
     from leaking through via their P31 chain)
   - Property path allows entities with a football-specific provider
     claim, with the non-football FILTER NOT EXISTS guard
   - FILTER NOT EXISTS { ?e wdt:P3450 ?parentComp } excludes seasonal
     entities (they're handled by the season query below)

   Matters because the old incremental discovery was picking up
   seasonal entities as competitions and letting non-football
   competitions leak through the property path.

Live-verified: the new team discovery query returns 98 entities
modified in the past day. Syntactically valid, non-zero, executes
against Wikidata's endpoint cleanly.

Note that this closes the drift for discovery, but the scoped per-QID
fetch query (build_scoped_ids_query) still uses VALUES to restrict —
which means entities that were modified BEFORE the expanded filter
went live still won't be discovered until a full refresh. For those,
the dump-based workflow (coming next) is the right tool.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: rahulkeerthi

scripts/incremental-update.py

@@ -97,7 +97,15 @@ def execute_sql_file(sql_path: str, remote: bool = True) -> bool:
 # ---------------------------------------------------------------------------
 
 def fetch_changed_qids(since: str) -> dict[str, list[str]]:
-    """Fetch QIDs modified since the given date, per entity type."""
+    """Fetch QIDs modified since the given date, per entity type.
+
+    These discovery queries must stay aligned with the initial-filter paths
+    in fetch-wikidata-entities.py's build_*_ids_query functions — any entity
+    reachable in the full fetch should also be discoverable here if it's
+    been edited recently. When the full-fetch filter widens, this file has
+    to widen too, otherwise newly-reachable entities would only arrive via
+    a full refresh.
+    """
     type_queries = {
         "player": f"""
             SELECT DISTINCT ?e WHERE {{
@@ -107,10 +115,27 @@ def fetch_changed_qids(since: str) -> dict[str, list[str]]:
               FILTER NOT EXISTS {{ ?e wdt:P31 wd:Q95074 }}
               FILTER NOT EXISTS {{ ?e wdt:P31 wd:Q15632617 }}
             }}""",
+        # Team: three-path UNION matching build_team_ids_query. Covers
+        # subclass of Q476028 (association football club), Q103229495
+        # (men's association football team — catches Q7156 Barcelona,
+        # Q170703 Boca), and sports clubs with football as sport.
         "team": f"""
             SELECT DISTINCT ?e WHERE {{
-              ?e wdt:P31 ?type .
-              ?type (wdt:P279)* wd:Q476028 .
+              {{
+                ?e wdt:P31 ?type .
+                ?type (wdt:P279)* wd:Q476028 .
+              }}
+              UNION
+              {{
+                ?e wdt:P31 ?type .
+                ?type (wdt:P279)* wd:Q103229495 .
+              }}
+              UNION
+              {{
+                ?e wdt:P641 wd:Q2736 .
+                ?e wdt:P31 ?type .
+                ?type (wdt:P279)* wd:Q847017 .
+              }}
               ?e schema:dateModified ?mod .
               FILTER(?mod > "{since}T00:00:00Z"^^xsd:dateTime)
             }}""",
@@ -122,15 +147,32 @@ def fetch_changed_qids(since: str) -> dict[str, list[str]]:
               FILTER NOT EXISTS {{ ?e wdt:P31 wd:Q95074 }}
               FILTER NOT EXISTS {{ ?e wdt:P31 wd:Q15632617 }}
             }}""",
+        # Competition: class path requires explicit P641 = football
+        # (blocks NHL / PGA / rugby leaks), property path allows any
+        # entity with a football-specific provider claim. Seasons are
+        # excluded via FILTER NOT EXISTS P3450 (they're handled by the
+        # season query below).
         "competition": f"""
             SELECT DISTINCT ?e WHERE {{
-              {{ ?e wdt:P31/wdt:P279* wd:Q15991290 . }}
-              UNION
-              {{ ?e wdt:P12758 [] . }}
-              UNION
-              {{ ?e wdt:P13664 [] . }}
-              UNION
-              {{ ?e wdt:P8735 [] . }}
+              {{
+                {{
+                  ?e wdt:P31/wdt:P279* wd:Q15991290 .
+                  ?e wdt:P641 wd:Q2736 .
+                }}
+                UNION
+                {{
+                  {{ ?e wdt:P12758 [] . }}
+                  UNION
+                  {{ ?e wdt:P13664 [] . }}
+                  UNION
+                  {{ ?e wdt:P8735 [] . }}
+                  FILTER NOT EXISTS {{
+                    ?e wdt:P641 ?sport .
+                    FILTER(?sport != wd:Q2736)
+                  }}
+                }}
+              }}
+              FILTER NOT EXISTS {{ ?e wdt:P3450 ?parentComp . }}
               ?e schema:dateModified ?mod .
               FILTER(?mod > "{since}T00:00:00Z"^^xsd:dateTime)
             }}""",