test: lock CI evidence contracts

Author: brycewang-stanford

.github/workflows/ci-cd.yml

@@ -90,16 +90,16 @@ jobs:
         python -m pip install --upgrade pip
         pip install -e ".[dev]"
 
-    - name: Install parity extra (canonical env — runs the doubleml external-parity pins)
-      # doubleml is the opt-in `parity` extra (not in `dev`). Installing it on
-      # the canonical ubuntu+3.10 env makes tests/external_parity/
-      # test_dml_python_parity.py actually run its 2 sp.dml-vs-doubleml pins,
-      # so the JSS external-parity headline count (52) is verified rather than
-      # silently skipped. Other matrix entries skip those 2 pins and the
+    - name: Install exact-parity extras (canonical env — runs DoubleML + CCT pins)
+      # doubleml and rdrobust are opt-in extras (not in `dev`). Installing them
+      # on the canonical ubuntu+3.10 env makes tests/external_parity/
+      # test_dml_python_parity.py run its 4 sp.dml-vs-doubleml pins and makes
+      # test_published_replications.py run the bwselect='cct' RD pins instead
+      # of skipping them. Other matrix entries may skip optional pins; the
       # count-drift guard degrades gracefully (see test_jss_validation_api.py).
       # Mirrors the flake8 / mypy / coverage canonical-env gating below.
       if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.10'
-      run: pip install -e ".[parity]"
+      run: pip install -e ".[parity,rd-cct]"
 
     - name: Lint with flake8 (hard syntax errors — all matrix entries)
       # E9/F63/F7/F82 are real syntax / undefined-name breakage and must be
@@ -130,11 +130,13 @@ jobs:
     # matrices and the same pin used by the flake8/mypy gates — where numpy is
     # >= 2.1 and the bug is absent. Every other matrix entry runs the full
     # suite without ``--cov``. Root cause: docs/rfc (E1 analysis).
-    # Coverage baseline: the 2026-05-28 local full-suite run (committed
-    # ``coverage.xml``, v1.16.0) measured 70.5% line coverage repo-wide
-    # (61,681 / 87,476 lines). The gate is therefore set to 60 — a 4x lift
-    # over the historical 15 placeholder, held ~10pp below the measured
-    # baseline so ordinary fold-split / platform variance cannot red a PR.
+    # Coverage baseline: the 2026-05-28 local full-suite coverage XML
+    # snapshot (v1.16.0) measured 70.5% line coverage repo-wide (61,681 /
+    # 87,476 lines). The gate is therefore set to 60 — a 4x lift over the
+    # historical 15 placeholder, held ~10pp below the measured baseline so
+    # ordinary fold-split / platform variance cannot red a PR. The current
+    # ratchet below consumes the CI-generated coverage.xml from this job; the
+    # root coverage.xml is a working artifact, not a tracked source of truth.
     # Ratchet upward (60 -> 65 -> 70) once each step is confirmed green in
     # CI on the canonical ubuntu+3.10 env; never set the gate above the last
     # CI-verified number.
@@ -146,8 +148,8 @@ jobs:
     # Per-module ratchet for the six core estimators (did iv rd synth dml panel).
     # The repo-wide --cov-fail-under above is a soft floor; this gate locks each
     # core module at the hard ≥95% the coverage campaign achieved so none can
-    # silently regress. Reuses the coverage.xml just produced. Ratchet the --min
-    # upward only after a higher number is confirmed green here.
+    # silently regress. Reuses the CI-generated coverage.xml just produced.
+    # Ratchet the --min upward only after a higher number is confirmed green here.
     - name: Core-module coverage ratchet (did/iv/rd/synth/dml/panel ≥ 95%)
       if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.10'
       run: |

docs/joss_reviewer_guide.md

@@ -33,7 +33,9 @@ are named `plotting`, `fixest`, `deepiv`, `neural`, `performance`, `bayes`,
 `tune`, `parity`, `rd-cct`, `text`, and `docs`. The `parity` extra adds the
 canonical Python DoubleML reference so the `sp.dml` machine-precision check
 runs instead of skipping (see *Double Machine Learning vs the DoubleML
-reference* below).
+reference* below). The `rd-cct` extra adds the official `rdrobust` Python port
+used by the exact CCT RD parity pins; the canonical CI job installs both
+`.[parity,rd-cct]`.
 
 ## Smoke Test
 

docs/joss_reviewer_qa.md

@@ -32,11 +32,12 @@ mechanisms make the breadth auditable rather than assertable:
    (`test_results_full_suite.md`).
 3. **Coverage is honest and rising.** The CI line-coverage gate was lifted
    from a 15% placeholder to **60%** against a measured **70.5%** baseline
-   (committed `coverage.xml`), and a dedicated, tracked campaign
-   (`.coverage_campaign/`) is driving the six core estimator modules
+   (historical full-suite coverage XML snapshot). The current CI gate uses the
+   `coverage.xml` generated inside the same job, while `.coverage_campaign/`
+   tracks the dedicated campaign driving the six core estimator modules
    (`did iv rd synth dml panel`) to **≥95%** line coverage under the full
-   suite — test-only, with real numerical assertions, no mocking of
-   numerical paths.
+   suite — test-only, with real numerical assertions, no mocking of numerical
+   paths.
 
 The honest framing — also stated in the validation dossier — is that
 *certified* methods carry parity or analytic evidence, while *breadth*

scripts/coverage_campaign.py

@@ -18,7 +18,8 @@
     python scripts/coverage_campaign.py gaps MODULE [--xml PATH] [--per-file]
 
 The default XML is ``.coverage_campaign/baseline_fullsuite.xml`` if present,
-else the repo-root committed ``coverage.xml``.
+else the repo-root ``coverage.xml`` generated by ``pytest --cov-report=xml``.
+The root XML is a working artifact, not a committed source of truth.
 
 This script is read-only over coverage data; it never runs pytest itself so it
 stays fast and side-effect free. Generate a fresh XML with, e.g.::

tests/test_ci_exact_parity_contract.py

@@ -0,0 +1,40 @@
+"""Static guards for optional exact-parity dependencies in CI."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def _read(path: str) -> str:
+    return (ROOT / path).read_text(encoding="utf-8")
+
+
+def test_canonical_ci_installs_all_exact_parity_extras():
+    workflow = _read(".github/workflows/ci-cd.yml")
+
+    assert "Install exact-parity extras" in workflow
+    assert "matrix.os == 'ubuntu-latest' && matrix.python-version == '3.10'" in workflow
+    assert 'pip install -e ".[parity,rd-cct]"' in workflow
+    assert "test_dml_python_parity.py run its 4 sp.dml-vs-doubleml pins" in workflow
+    assert "test_published_replications.py run the bwselect='cct' RD pins" in workflow
+
+
+def test_optional_dependency_names_match_pyproject_extras_and_skips():
+    pyproject = _read("pyproject.toml")
+    dml_test = _read("tests/external_parity/test_dml_python_parity.py")
+    published_test = _read("tests/external_parity/test_published_replications.py")
+
+    assert "parity = [" in pyproject
+    assert '"doubleml>=0.7"' in pyproject
+    assert "rd-cct = [" in pyproject
+    assert '"rdrobust>=1.3"' in pyproject
+
+    assert 'pytest.importorskip("doubleml")' in dml_test
+    assert "all four DoubleML model classes" in dml_test
+
+    assert "pytest.importorskip(\n            'rdrobust'," in published_test
+    assert "extras [rd-cct] are not installed" in published_test
+    assert r"pip install statspai\[rd-cct\]" in published_test

tests/test_coverage_evidence_contract.py

@@ -0,0 +1,49 @@
+"""Coverage evidence must be generated live, not treated as stale fixtures."""
+
+from __future__ import annotations
+
+import subprocess
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def _read(path: str) -> str:
+    return (ROOT / path).read_text(encoding="utf-8")
+
+
+def test_root_coverage_xml_is_not_a_committed_evidence_artifact():
+    proc = subprocess.run(
+        ["git", "ls-files", "coverage.xml"],
+        cwd=ROOT,
+        check=True,
+        capture_output=True,
+        text=True,
+    )
+    assert proc.stdout.strip() == ""
+
+
+def test_ci_coverage_ratchet_consumes_same_job_generated_xml():
+    workflow = _read(".github/workflows/ci-cd.yml")
+    coverage_step = (
+        "pytest tests/ -v --cov=statspai --cov-report=xml "
+        "--cov-report=term-missing --cov-fail-under=60"
+    )
+    ratchet_step = "python scripts/coverage_campaign.py report --xml coverage.xml --check --min 95"
+
+    assert coverage_step in workflow
+    assert ratchet_step in workflow
+    assert workflow.index(coverage_step) < workflow.index(ratchet_step)
+    assert "CI-generated coverage.xml just produced" in workflow
+
+
+def test_coverage_campaign_and_docs_do_not_call_root_xml_committed_truth():
+    script = _read("scripts/coverage_campaign.py")
+    reviewer_qa = _read("docs/joss_reviewer_qa.md")
+
+    assert "committed ``coverage.xml``" not in script
+    assert "committed `coverage.xml`" not in reviewer_qa
+    assert "working artifact, not a committed source of truth" in script
+    assert "The current CI gate uses the" in reviewer_qa
+    assert "`coverage.xml` generated inside the same job" in reviewer_qa

tests/test_jss_validation_api.py

@@ -193,8 +193,9 @@ def test_validation_report_collected_counts_match_jss_headline():
             # The manuscript's external-parity count includes the 4
             # doubleml-gated pins; without the optional `parity` extra they
             # don't collect, so verifying the floor here would be a spurious
-            # failure. CI installs `.[parity]` on the canonical env to check
-            # the full count; skip it in minimal environments instead.
+            # failure. CI installs `.[parity,rd-cct]` on the canonical env to
+            # check the full count and to run the CCT RD pins; skip it in
+            # minimal environments instead.
             continue
         assert actual >= expected, (
             f"{key}: collected {actual} tests, which is BELOW the JSS "