Skip to content

Commit 9b5eb6e

Browse files
feat(did): xvar heterogeneity + parity tests + custom summary + did_report
Four-in-one DID polish round, all scoped to the DID module: 1. etwfe(xvar=...) — covariate-moderated heterogeneity New _etwfe_with_xvar helper adds cohort × post × (x - mean(x)) interactions so the baseline coefficient is ATT(g) evaluated at the sample mean of xvar and the slope coefficient measures how ATT(g) shifts per unit of xvar. Closes the last R-etwfe feature gap. Backward compatible: etwfe(xvar=None) is byte-identical to wooldridge_did(). 2. tests/test_did_summary.py — DID consistency + parity tests Seven pytest cases covering: did_summary ↔ direct etwfe / BJS numeric match; etwfe ≡ wooldridge_did under xvar=None; xvar= random-covariate sanity bound; Markdown + LaTeX exports shape + content; non-did_summary input correctly rejected; forest plot render via Agg backend. pytest exit 0 (7 passed in 6.34s). 3. Custom result.summary() on did_summary output Prints a Markdown table (via did_summary_to_markdown) plus a header block with fitted/failed methods, cross-method mean, dispersion SD, and breakdown M* (when sensitivity was run). Base CausalResult.summary's single-estimate layout was a poor fit for a multi-method comparison; this customisation gives researchers a paste-ready console dump. 4. did_report(data, ..., save_to=dir) — one-call bundle Fits did_summary with include_sensitivity=True by default and writes five files into save_to/: - did_summary.txt (summary dump) - did_summary.md (GFM table) - did_summary.tex (booktabs LaTeX fragment) - did_summary.png (forest plot, matplotlib-optional) - did_summary.json (detail + model_info, replay-friendly) Returns the underlying CausalResult for programmatic use. READMEs updated (EN + CN). Commit staged with explicit file list to avoid the earlier `git add -A` sweep regression. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8d97da2 commit 9b5eb6e

7 files changed

Lines changed: 450 additions & 12 deletions

File tree

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,8 @@ If a method exists in R, we aim to match or exceed its feature set in Python —
167167
| `did()` | Auto-dispatching DID (2×2 or staggered) ||
168168
| `did_summary()` | One-call robustness comparison across CS/SA/BJS/ETWFE/Stacked ||
169169
| `did_summary_plot()` | Forest plot of method-robustness summary ||
170+
| `did_summary_to_markdown()` / `_to_latex()` | Publication-ready tables from `did_summary` ||
171+
| `did_report()` | One-call bundle: txt + md + tex + png + json into a folder ||
170172
| `did_2x2()` | Classic two-group, two-period DID ||
171173
| `callaway_santanna()` | Staggered DID with heterogeneous effects | Callaway & Sant'Anna (2021) |
172174
| `sun_abraham()` | Interaction-weighted event study | Sun & Abraham (2021) |

README_CN.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,8 @@ StatsPAI **不是** R 的 wrapper。我们从原始论文独立重新实现每
8686
| `did()` | 自动分派 DID(2×2 或交错) ||
8787
| `did_summary()` | 一次调用跑 CS/SA/BJS/ETWFE/Stacked 五种方法并出对比表 ||
8888
| `did_summary_plot()` | 方法稳健性森林图(点估计 + CI 并排) ||
89+
| `did_summary_to_markdown()` / `_to_latex()` | 论文级稳健性对比表(GFM / booktabs) ||
90+
| `did_report()` | 一次调用打包输出:txt + md + tex + png + json ||
8991
| `callaway_santanna()` | 交错 DID,异质处理效应 | Callaway & Sant'Anna (2021) |
9092
| `sun_abraham()` | 交互加权事件研究 | Sun & Abraham (2021) |
9193
| `bacon_decomposition()` | TWFE 分解诊断 | Goodman-Bacon (2021) |

src/statspai/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
bacon_decomposition, honest_did, breakdown_m, event_study,
3636
did_analysis, DIDAnalysis, did_multiplegt, did_imputation, stacked_did, cic,
3737
wooldridge_did, etwfe, drdid, twfe_decomposition,
38-
did_summary, did_summary_to_markdown, did_summary_to_latex,
38+
did_summary, did_summary_to_markdown, did_summary_to_latex, did_report,
3939
pretrends_test, pretrends_power, sensitivity_rr, SensitivityResult, pretrends_summary,
4040
parallel_trends_plot, bacon_plot, group_time_plot, did_plot,
4141
enhanced_event_study_plot, treatment_rollout_plot,
@@ -231,6 +231,7 @@
231231
"did_summary",
232232
"did_summary_to_markdown",
233233
"did_summary_to_latex",
234+
"did_report",
234235
"drdid",
235236
"twfe_decomposition",
236237
# RD

src/statspai/did/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
pretrends_summary,
4747
)
4848
from .wooldridge_did import wooldridge_did, etwfe, drdid, twfe_decomposition
49-
from .summary import did_summary, did_summary_to_markdown, did_summary_to_latex
49+
from .summary import did_summary, did_summary_to_markdown, did_summary_to_latex, did_report
5050
from .continuous_did import continuous_did
5151
from .plots import (
5252
parallel_trends_plot,
@@ -390,6 +390,7 @@ def did(
390390
'did_summary',
391391
'did_summary_to_markdown',
392392
'did_summary_to_latex',
393+
'did_report',
393394
'drdid',
394395
'twfe_decomposition',
395396
# Pre-trends

src/statspai/did/summary.py

Lines changed: 137 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -303,7 +303,7 @@ def did_summary(
303303
else:
304304
avg_est, disp = np.nan, np.nan
305305

306-
return CausalResult(
306+
result = CausalResult(
307307
method="DID Method-Robustness Summary",
308308
estimand="Overall ATT (mean across methods)",
309309
estimate=avg_est,
@@ -323,6 +323,28 @@ def did_summary(
323323
_citation_key="did_summary",
324324
)
325325

326+
# Custom .summary() — prints the comparison table as a Markdown block
327+
# instead of the base CausalResult's single-estimate format.
328+
def _custom_summary(alpha=None): # noqa: ARG001
329+
header = "=" * 78 + "\n"
330+
header += " DID Method-Robustness Summary\n"
331+
header += "=" * 78 + "\n\n"
332+
if fit:
333+
header += f" Fitted methods: {', '.join(fit)}\n"
334+
if failed:
335+
header += f" Failed: {', '.join(failed.keys())}\n"
336+
if not np.isnan(avg_est):
337+
header += f" Mean ATT: {avg_est:+.4f}\n"
338+
header += f" Cross-method SD: {disp:+.4f}\n"
339+
if breakdown_m_value is not None:
340+
header += f" Breakdown M*: {breakdown_m_value:.4f} (CS row)\n"
341+
header += "\n"
342+
body = did_summary_to_markdown(result)
343+
return header + body
344+
345+
result.summary = _custom_summary
346+
return result
347+
326348

327349
# ═══════════════════════════════════════════════════════════════════════
328350
# Export helpers: publication-ready Markdown / LaTeX from a did_summary
@@ -521,3 +543,117 @@ def did_summary_to_latex(
521543
)
522544
lines.append("\\end{table}")
523545
return "\n".join(lines)
546+
547+
548+
# ═══════════════════════════════════════════════════════════════════════
549+
# One-call DID bundle report — txt + md + tex + png in a single directory
550+
# ═══════════════════════════════════════════════════════════════════════
551+
552+
def did_report(
553+
data: pd.DataFrame,
554+
y: str,
555+
time: str,
556+
first_treat: str,
557+
group: str,
558+
save_to: str,
559+
methods: Union[str, List[str]] = "auto",
560+
controls: Optional[List[str]] = None,
561+
cluster: Optional[str] = None,
562+
alpha: float = 0.05,
563+
include_sensitivity: bool = True,
564+
plot_sort_by: Optional[str] = "estimate",
565+
verbose: bool = False,
566+
) -> CausalResult:
567+
"""
568+
One-call comprehensive DID report: fits all methods + exports every format.
569+
570+
Writes the following files to ``save_to``:
571+
572+
- ``did_summary.txt`` : text dump of ``result.summary()``.
573+
- ``did_summary.md`` : GitHub-Flavoured Markdown table.
574+
- ``did_summary.tex`` : publication-ready LaTeX ``booktabs`` fragment.
575+
- ``did_summary.png`` : forest plot (requires matplotlib).
576+
- ``did_summary.json`` : detail table + model_info in JSON.
577+
578+
Parameters
579+
----------
580+
data, y, time, first_treat, group, methods, controls, cluster, alpha
581+
Same as :func:`did_summary`.
582+
save_to : str
583+
Directory path. Created if it does not exist.
584+
include_sensitivity : bool, default True
585+
Whether to run Rambachan-Roth breakdown M*. Defaults to ``True``
586+
in ``did_report`` (vs ``False`` in ``did_summary``) because a
587+
report is expected to be comprehensive.
588+
plot_sort_by : {'estimate', None}, default 'estimate'
589+
Sort the forest plot by estimate ascending.
590+
verbose : bool, default False
591+
Print progress lines.
592+
593+
Returns
594+
-------
595+
CausalResult
596+
The underlying :func:`did_summary` output. All side-effect files
597+
are written to ``save_to`` as a bundle.
598+
"""
599+
import json
600+
from pathlib import Path
601+
602+
out_dir = Path(save_to)
603+
out_dir.mkdir(parents=True, exist_ok=True)
604+
605+
if verbose:
606+
print(f"[did_report] fitting methods into {out_dir} ...", flush=True)
607+
608+
result = did_summary(
609+
data, y=y, time=time, first_treat=first_treat, group=group,
610+
methods=methods, controls=controls, cluster=cluster, alpha=alpha,
611+
include_sensitivity=include_sensitivity, verbose=verbose,
612+
)
613+
614+
# TXT
615+
(out_dir / "did_summary.txt").write_text(result.summary(), encoding="utf-8")
616+
617+
# Markdown
618+
(out_dir / "did_summary.md").write_text(
619+
did_summary_to_markdown(result), encoding="utf-8",
620+
)
621+
622+
# LaTeX
623+
(out_dir / "did_summary.tex").write_text(
624+
did_summary_to_latex(result), encoding="utf-8",
625+
)
626+
627+
# PNG (best-effort — skip if matplotlib unavailable)
628+
try:
629+
from .plots import did_summary_plot
630+
fig, _ = did_summary_plot(result, sort_by=plot_sort_by)
631+
fig.savefig(out_dir / "did_summary.png", dpi=150, bbox_inches="tight")
632+
try:
633+
import matplotlib.pyplot as plt
634+
plt.close(fig)
635+
except Exception:
636+
pass
637+
except ImportError:
638+
if verbose:
639+
print("[did_report] matplotlib missing — skipping PNG", flush=True)
640+
641+
# JSON (detail + model_info)
642+
payload = {
643+
"method": result.method,
644+
"estimate": None if pd.isna(result.estimate) else float(result.estimate),
645+
"cross_method_sd": None if pd.isna(result.se) else float(result.se),
646+
"detail": result.detail.replace({np.nan: None}).to_dict(orient="records"),
647+
"model_info": {
648+
k: (list(v) if isinstance(v, (set, tuple)) else v)
649+
for k, v in result.model_info.items()
650+
},
651+
}
652+
(out_dir / "did_summary.json").write_text(
653+
json.dumps(payload, indent=2, default=str), encoding="utf-8",
654+
)
655+
656+
if verbose:
657+
print(f"[did_report] wrote bundle to {out_dir}", flush=True)
658+
659+
return result

src/statspai/did/wooldridge_did.py

Lines changed: 156 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -358,6 +358,7 @@ def etwfe(
358358
controls: Optional[List[str]] = None,
359359
cluster: Optional[str] = None,
360360
alpha: float = 0.05,
361+
xvar: Optional[str] = None,
361362
) -> CausalResult:
362363
"""
363364
Extended Two-Way Fixed Effects (ETWFE) — Wooldridge (2021).
@@ -403,7 +404,7 @@ def etwfe(
403404
``tvar = time`` ``time='time'``
404405
``gvar = first_treat`` ``first_treat='first_treat'``
405406
``ivar = unit`` ``group='unit'``
406-
``xvar`` (covariate het.) *not yet supported — use* ``controls``
407+
``xvar`` (covariate het.) ``xvar='x1'`` (single covariate)
407408
``vcov = ~cluster`` ``cluster='cluster'``
408409
============================ ========================================
409410
@@ -433,15 +434,161 @@ def etwfe(
433434
callaway_santanna : CS (2021) group-time ATT estimator.
434435
aggte : Aggregation of group-time ATTs (event/group/calendar/simple).
435436
"""
436-
return wooldridge_did(
437-
data=data,
438-
y=y,
439-
group=group,
440-
time=time,
441-
first_treat=first_treat,
442-
controls=controls,
443-
cluster=cluster,
437+
if xvar is None:
438+
return wooldridge_did(
439+
data=data, y=y, group=group, time=time, first_treat=first_treat,
440+
controls=controls, cluster=cluster, alpha=alpha,
441+
)
442+
return _etwfe_with_xvar(
443+
data=data, y=y, group=group, time=time, first_treat=first_treat,
444+
xvar=xvar, controls=controls, cluster=cluster, alpha=alpha,
445+
)
446+
447+
448+
def _etwfe_with_xvar(
449+
data: pd.DataFrame,
450+
y: str,
451+
group: str,
452+
time: str,
453+
first_treat: str,
454+
xvar: str,
455+
controls: Optional[List[str]] = None,
456+
cluster: Optional[str] = None,
457+
alpha: float = 0.05,
458+
) -> CausalResult:
459+
"""ETWFE with covariate-moderated heterogeneity (R etwfe's ``xvar``).
460+
461+
For each cohort ``g``, adds an interaction between the
462+
cohort × post dummy and ``(xvar - mean(xvar))``, so the slope
463+
coefficient measures how ATT(g) shifts per unit of ``xvar``. The
464+
main cohort coefficient is then interpretable as ATT(g) evaluated
465+
at the sample mean of ``xvar``.
466+
"""
467+
df = data.copy()
468+
ft = df[first_treat].replace(0, np.nan)
469+
df["_ft"] = ft
470+
471+
periods = sorted(df[time].unique())
472+
cohorts = sorted(df.loc[df["_ft"].notna(), "_ft"].unique())
473+
if len(cohorts) == 0:
474+
raise ValueError("No treated cohorts found. Check 'first_treat' column.")
475+
476+
# Demean outcome
477+
df["_y"] = df[y].astype(float)
478+
unit_mean = df.groupby(group)["_y"].transform("mean")
479+
time_mean = df.groupby(time)["_y"].transform("mean")
480+
grand_mean = df["_y"].mean()
481+
df["_y_dm"] = df["_y"] - unit_mean - time_mean + grand_mean
482+
483+
# Center xvar by grand mean so the baseline ATT is evaluated at x=mean
484+
df["_x"] = df[xvar].astype(float)
485+
x_center = df["_x"].mean()
486+
df["_xc"] = df["_x"] - x_center
487+
488+
# Build cohort × post dummies AND cohort × post × xc interactions
489+
base_cols: List[str] = []
490+
slope_cols: List[str] = []
491+
for g in cohorts:
492+
mask_post = (df["_ft"] == g) & (df[time] >= g)
493+
b = f"_coh{int(g)}_post"
494+
s = f"_coh{int(g)}_post_x"
495+
df[b] = mask_post.astype(float)
496+
df[s] = df[b] * df["_xc"]
497+
base_cols.append(b)
498+
slope_cols.append(s)
499+
500+
all_inter = base_cols + slope_cols
501+
# Demean interactions via FE projection
502+
for col in all_inter:
503+
u_m = df.groupby(group)[col].transform("mean")
504+
t_m = df.groupby(time)[col].transform("mean")
505+
g_m = df[col].mean()
506+
df[f"{col}_dm"] = df[col] - u_m - t_m + g_m
507+
508+
# Demean controls
509+
ctrl_dm_cols: List[str] = []
510+
if controls:
511+
for c in controls:
512+
df[f"_ctrl_{c}"] = df[c].astype(float)
513+
u_m = df.groupby(group)[f"_ctrl_{c}"].transform("mean")
514+
t_m = df.groupby(time)[f"_ctrl_{c}"].transform("mean")
515+
g_m = df[f"_ctrl_{c}"].mean()
516+
df[f"_ctrl_{c}_dm"] = df[f"_ctrl_{c}"] - u_m - t_m + g_m
517+
ctrl_dm_cols.append(f"_ctrl_{c}_dm")
518+
519+
keep = ["_y_dm"] + [f"{c}_dm" for c in all_inter] + ctrl_dm_cols
520+
valid = df[keep].notna().all(axis=1)
521+
dfv = df.loc[valid].reset_index(drop=True)
522+
523+
y_vec = dfv["_y_dm"].values
524+
X_cols = [f"{c}_dm" for c in all_inter] + ctrl_dm_cols
525+
X = dfv[X_cols].values
526+
X = np.column_stack([np.ones(len(y_vec)), X])
527+
528+
cl_arr = dfv[cluster].values if cluster else dfv[group].values
529+
beta, se, vcov = _ols_fit(X, y_vec, cluster=cl_arr)
530+
n_obs = len(y_vec)
531+
df_resid = n_obs - X.shape[1]
532+
533+
k = len(cohorts)
534+
# Baseline ATT(g) at x=mean: coefficients 1..k
535+
# Slope(g): coefficients k+1..2k
536+
cohort_results = []
537+
for i, g in enumerate(cohorts):
538+
b_idx = 1 + i
539+
s_idx = 1 + k + i
540+
att = float(beta[b_idx])
541+
att_se = float(se[b_idx])
542+
slope = float(beta[s_idx])
543+
slope_se = float(se[s_idx])
544+
p = float(2 * (1 - stats.t.cdf(abs(att / att_se) if att_se > 0 else 0,
545+
max(df_resid, 1))))
546+
p_slope = float(2 * (1 - stats.t.cdf(abs(slope / slope_se) if slope_se > 0 else 0,
547+
max(df_resid, 1))))
548+
n_g = int((dfv["_ft"] == g).sum())
549+
cohort_results.append({
550+
"cohort": int(g),
551+
"att_at_xmean": att, "att_se": att_se, "att_pvalue": p,
552+
"slope_wrt_x": slope, "slope_se": slope_se, "slope_pvalue": p_slope,
553+
"n_obs": n_g,
554+
})
555+
556+
detail = pd.DataFrame(cohort_results)
557+
sizes = detail["n_obs"].values.astype(float)
558+
weights_vec = sizes / sizes.sum() if sizes.sum() > 0 else np.ones(k) / k
559+
att_overall = float(weights_vec @ detail["att_at_xmean"].values)
560+
# SE via delta method using vcov of baseline coefficients
561+
base_vcov = vcov[1:1 + k, 1:1 + k]
562+
att_se_overall = float(np.sqrt(weights_vec @ base_vcov @ weights_vec))
563+
t_stat = att_overall / att_se_overall if att_se_overall > 0 else np.nan
564+
p_overall = float(2 * (1 - stats.t.cdf(abs(t_stat), max(df_resid, 1))))
565+
t_crit = stats.t.ppf(1 - alpha / 2, max(df_resid, 1))
566+
ci = (att_overall - t_crit * att_se_overall, att_overall + t_crit * att_se_overall)
567+
568+
model_info = {
569+
"n_cohorts": k,
570+
"cohorts": [int(g) for g in cohorts],
571+
"n_periods": len(periods),
572+
"n_units": df[group].nunique(),
573+
"controls": controls or [],
574+
"cluster_var": cluster or group,
575+
"xvar": xvar,
576+
"xvar_mean": float(x_center),
577+
"heterogeneity": "ATT(g) = baseline(g) + slope(g) * (xvar - mean(xvar))",
578+
}
579+
580+
return CausalResult(
581+
method="Wooldridge (2021) ETWFE with covariate heterogeneity",
582+
estimand=f"ATT at {xvar} = {x_center:.4g} (sample mean)",
583+
estimate=att_overall,
584+
se=att_se_overall,
585+
pvalue=p_overall,
586+
ci=ci,
444587
alpha=alpha,
588+
n_obs=n_obs,
589+
detail=detail,
590+
model_info=model_info,
591+
_citation_key="wooldridge_twfe",
445592
)
446593

447594

0 commit comments

Comments
 (0)