|
| 1 | +--- |
| 2 | +title: "feat: Replace custom TIGER/Line downloads with pygris" |
| 3 | +type: feat |
| 4 | +status: active |
| 5 | +date: 2026-03-13 |
| 6 | +--- |
| 7 | + |
| 8 | +# feat: Replace custom TIGER/Line downloads with pygris |
| 9 | + |
| 10 | +## Overview |
| 11 | + |
| 12 | +Replace pypums's hand-rolled TIGER/Line cartographic boundary download code in `pypums/spatial.py` with the [pygris](https://github.com/walkerke/pygris) package. pygris is the Python port of R's `tigris` — a well-maintained library purpose-built for downloading Census Bureau shapefiles. This removes ~40 lines of URL-construction logic, gains built-in caching, better year/vintage handling, and broader geography support for free. |
| 13 | + |
| 14 | +## Problem Statement / Motivation |
| 15 | + |
| 16 | +The current `_fetch_tiger_shapes()` function manually constructs Census Bureau URLs from templates and calls `geopandas.read_file(url)`. This works but: |
| 17 | + |
| 18 | +- **Fragile URL construction** — hardcoded URL patterns break when the Census Bureau changes paths across vintages (e.g., ZCTA and PUMA vintage suffixes are hardcoded to 2020, broken for pre-2020 years) |
| 19 | +- **No caching** — every `geometry=True` call re-downloads the same shapefile |
| 20 | +- **Limited year support** — only works for the modern `GENZ{year}/shp/` URL pattern (roughly 2014+) |
| 21 | +- **Reinvents the wheel** — pygris already solves all of this with a maintained, tested codebase |
| 22 | + |
| 23 | +## Proposed Solution |
| 24 | + |
| 25 | +Replace the internals of `_fetch_tiger_shapes()` with calls to the corresponding pygris functions. Keep the public API (`attach_geometry()`, `geometry=True` flag on data functions) completely unchanged. |
| 26 | + |
| 27 | +### Geography mapping |
| 28 | + |
| 29 | +| pypums geography | pygris function | Notes | |
| 30 | +|---|---|---| |
| 31 | +| `"state"` | `pygris.states(cb=True)` | `cb` defaults True for states already | |
| 32 | +| `"county"` | `pygris.counties(state=..., cb=True)` | Pass `state` through for perf | |
| 33 | +| `"tract"` | `pygris.tracts(state=..., cb=True)` | `state` required | |
| 34 | +| `"block group"` | `pygris.block_groups(state=..., cb=True)` | `state` required | |
| 35 | +| `"place"` | `pygris.places(state=..., cb=True)` | | |
| 36 | +| `"congressional district"` | `pygris.congressional_districts(cb=True)` | pygris handles congress number internally | |
| 37 | +| `"zcta"` | `pygris.zctas(cb=True)` | pygris handles vintage suffix | |
| 38 | +| `"puma"` | `pygris.pumas(state=..., cb=True)` | pygris handles vintage suffix | |
| 39 | +| `"cbsa"` | `pygris.core_based_statistical_areas(cb=True)` | | |
| 40 | +| `"csa"` | `pygris.combined_statistical_areas(cb=True)` | | |
| 41 | + |
| 42 | +### Implementation approach |
| 43 | + |
| 44 | +1. **Preserve `_fetch_tiger_shapes` as a thin wrapper** — replace `_GEO_TO_TIGER` dict with a `_GEO_TO_PYGRIS` dict that maps geography names to pygris callables. The function dispatches to the right pygris function with `cb=True`, `year`, `resolution`, and `state` as applicable. |
| 45 | + |
| 46 | +2. **Normalize GEOID column** — after receiving shapes from pygris, ensure the `GEOID` column exists (pygris may return `GEOID20`, `GEOID10` for some vintages). Add a small normalization step. |
| 47 | + |
| 48 | +3. **Pass `state` through to pygris** for sub-state and county-level geographies to download smaller files (faster). The right-join merge already filters to matching GEOIDs, so results are identical. |
| 49 | + |
| 50 | +4. **Remove dead constants** — delete `_TIGER_BASE` and `_GEO_TO_TIGER` after replacement. |
| 51 | + |
| 52 | +5. **Add `pygris` to the `spatial` optional dependency group** in `pyproject.toml` and run `uv lock`. |
| 53 | + |
| 54 | +## Technical Considerations |
| 55 | + |
| 56 | +- **CRS contract** — docs promise EPSG:4269 (NAD83). Add an assertion or `.to_crs()` call after pygris returns to guarantee this. |
| 57 | +- **Caching** — pygris caches shapefiles to `~/Library/Caches/pygris/` (macOS) / `~/.cache/pygris/` (Linux) when `cache=True`. We should pass `cache=True` by default to avoid re-downloading on every call — this is a UX improvement over the current behavior. |
| 58 | +- **Error handling** — keep the `ValueError` for unsupported geographies. Let pygris errors for year/network issues propagate naturally; they are descriptive enough. |
| 59 | +- **Test mocks** — preserving `_fetch_tiger_shapes` means the existing mock target `"pypums.spatial._fetch_tiger_shapes"` in `tests/test_spatial.py` continues to work unchanged. |
| 60 | +- **Network stack change** — pygris uses `requests` instead of geopandas/fiona's urllib. This is generally more robust but could differ for users behind corporate proxies. Low risk. |
| 61 | +- **Congressional districts** — pygris handles the congress-number-to-year mapping internally, so we can drop the manual `113 + (year - 2013) // 2` computation. |
| 62 | +- **ZCTA/PUMA vintages** — the current code hardcodes 2020 vintage suffixes (`zcta520`, `puma20`), which is broken for pre-2020 years. pygris handles this correctly — a free bug fix. |
| 63 | + |
| 64 | +## Acceptance Criteria |
| 65 | + |
| 66 | +- [x] `_fetch_tiger_shapes` calls pygris functions instead of constructing URLs manually |
| 67 | +- [x] `_GEO_TO_TIGER` and `_TIGER_BASE` constants are removed |
| 68 | +- [x] New `_GEO_TO_PYGRIS` mapping covers all 10 geography levels |
| 69 | +- [x] GEOID column is normalized after pygris returns (handle `GEOID`, `GEOID20`, `GEOID10` variants) |
| 70 | +- [x] `pygris` added to `spatial` optional deps in `pyproject.toml` |
| 71 | +- [x] `uv lock` run and `uv.lock` updated |
| 72 | +- [x] `resolution` parameter still works on `attach_geometry()` and maps to pygris `resolution` |
| 73 | +- [x] `state` passed through to pygris for county/sub-state geographies |
| 74 | +- [x] CRS assertion added (EPSG:4269) |
| 75 | +- [x] Caching enabled by default (`cache=True`) |
| 76 | +- [x] Existing tests pass without changes to test code (mock target preserved) |
| 77 | +- [x] Module and function docstrings updated to mention pygris |
| 78 | +- [x] `docs/guides/spatial.md` updated: mention pygris, caching, installation changes |
| 79 | +- [x] Branch: `feat/use-pygris` created from `main` |
| 80 | +- [ ] PR created targeting `main` |
| 81 | + |
| 82 | +## Success Metrics |
| 83 | + |
| 84 | +- All existing tests pass (`uv run pytest`) |
| 85 | +- `geometry=True` works for all 10 geography levels (manual spot check) |
| 86 | +- Repeated calls are faster due to caching |
| 87 | +- Docs build cleanly (`uv run mkdocs build --strict`) |
| 88 | + |
| 89 | +## Dependencies & Risks |
| 90 | + |
| 91 | +- **pygris version** — pin to `>=0.1.7` (last version with all 10 geography functions). Current latest is 0.2.1. |
| 92 | +- **pygris maintenance** — actively maintained by Kyle Walker (author of R tigris/tidycensus). Low bus-factor risk. |
| 93 | +- **New transitive deps** — `requests` and `platformdirs` (both widely used, low risk) |
| 94 | +- **Breaking change risk** — low. Public API is unchanged. Only internal implementation changes. The ZCTA/PUMA fix for pre-2020 years is technically a behavior change but corrects a bug. |
| 95 | + |
| 96 | +## Implementation Steps |
| 97 | + |
| 98 | +### Phase 1: Core replacement (`pypums/spatial.py`) |
| 99 | + |
| 100 | +1. Create branch `feat/use-pygris` from `main` |
| 101 | +2. Add `pygris>=0.1.7` to `[project.optional-dependencies] spatial` in `pyproject.toml` |
| 102 | +3. Run `uv lock` |
| 103 | +4. Replace `_GEO_TO_TIGER` dict with `_GEO_TO_PYGRIS` mapping (geography name -> pygris callable) |
| 104 | +5. Rewrite `_fetch_tiger_shapes` to dispatch to pygris functions with `cb=True`, `year`, `resolution`, `state` |
| 105 | +6. Add GEOID column normalization |
| 106 | +7. Add CRS assertion |
| 107 | +8. Remove `_TIGER_BASE` and `_GEO_TO_TIGER` |
| 108 | +9. Update module docstring and function docstrings |
| 109 | + |
| 110 | +### Phase 2: Tests & docs |
| 111 | + |
| 112 | +10. Run `uv run pytest` — existing tests should pass (mock target preserved) |
| 113 | +11. Update `docs/guides/spatial.md`: |
| 114 | + - Update "How it works" to mention pygris |
| 115 | + - Add note about automatic caching |
| 116 | + - Update installation section if needed |
| 117 | + - Update troubleshooting section |
| 118 | +12. Run `uv run mkdocs build --strict` to verify docs |
| 119 | +13. Run `uv run ruff check --fix . && uv run ruff format .` |
| 120 | + |
| 121 | +### Phase 3: PR & release |
| 122 | + |
| 123 | +14. Commit all changes |
| 124 | +15. Push branch and create PR |
| 125 | +16. After merge, release as v0.3.1 |
| 126 | + |
| 127 | +## Sources & References |
| 128 | + |
| 129 | +- [pygris GitHub](https://github.com/walkerke/pygris) |
| 130 | +- [pygris docs — basic usage](https://walker-data.com/pygris/01-basic-usage/) |
| 131 | +- [pygris on PyPI](https://pypi.org/project/pygris/) |
| 132 | +- Current implementation: `pypums/spatial.py` |
| 133 | +- Tests: `tests/test_spatial.py` |
| 134 | +- Docs: `docs/guides/spatial.md` |
| 135 | +- Callers: `pypums/acs.py:233`, `pypums/decennial.py:175`, `pypums/estimates.py:266`, `pypums/flows.py:251` |
0 commit comments