Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# CLAUDE.md

Project-level context for Claude Code sessions working on PyPUMS.

## Project overview

PyPUMS is a Python interface to the US Census Bureau API. It provides functions for
ACS, Decennial Census, PUMS microdata, population estimates, and migration flows.
The package returns pandas DataFrames (or GeoDataFrames when `geometry=True`).

- **Package source:** `pypums/`
- **Tests:** `tests/`
- **Docs:** `docs/` (MkDocs Material, hosted on ReadTheDocs)
- **CLI:** `pypums/cli.py` (Typer)

## Common commands

```bash
# Run tests (uses doctest + pytest)
uv run pytest

# Run a specific test file
uv run pytest tests/test_get_acs.py

# Lint and format
uv run ruff check --fix .
uv run ruff format .

# Build docs locally
uv run mkdocs build --strict

# Serve docs locally (live reload)
uv run mkdocs serve
```

## Dependencies and lockfile

This project uses **uv** for dependency management.

**Important:** When you add or change dependencies in `pyproject.toml`, you **must**
run `uv lock` and commit the updated `uv.lock`. ReadTheDocs uses
`uv sync --frozen --extra docs` which requires the lockfile to be in sync with
`pyproject.toml`. Forgetting to update `uv.lock` will cause the docs build to fail
on CI.

Optional dependency groups:
- `spatial` — geopandas (for `geometry=True` support)
- `test` — pytest
- `docs` — mkdocs-material, mkdocstrings, mkdocs-charts-plugin, altair, etc.

## Code style

- **Linter/formatter:** Ruff (config in `pyproject.toml`)
- **Target:** Python 3.10+
- **Pre-commit hooks:** trailing whitespace, end-of-file fixer, check-yaml, check-toml,
ruff-check, ruff-format

## Docs

The documentation uses **MkDocs Material** with these notable plugins/extensions:

- `mkdocs-charts-plugin` with `pymdownx.superfences` custom fences for rendering
**Vega-Lite** charts inline. Use ` ```vegalite ` fenced code blocks with a JSON
Vega-Lite spec to render interactive charts directly in the docs.
- `mkdocstrings[python]` for API reference (numpy-style docstrings).
- `mkdocs-typer` for CLI reference auto-generation.
- Visualizations use **Altair** (not matplotlib). Code examples should use Altair, and
rendered previews use Vega-Lite JSON specs in `vegalite` fenced blocks.

Rendered Vega-Lite charts in docs should:
- Use inline `"values"` data (not live API calls) so they render without a backend.
- Include all 50 states + DC for choropleth maps.
- Be wrapped in `!!! example "Interactive preview"` admonitions when the rendered chart
shows a different geographic scale than the code example (e.g., state-level preview
for a tract-level code example).

## CI pipeline

- **GitHub Actions:** lint (ruff), tests (Python 3.10-3.13)
- **ReadTheDocs:** builds docs from `.readthedocs.yaml` using `uv sync --frozen --extra docs`
- **Pre-commit hooks:** run on every commit (trailing whitespace, ruff, etc.)

## Test markers

```
phase0 — Foundation (API infra, geography, FIPS, cache)
phase1 — Core data functions (get_acs, get_decennial, load_variables)
phase2 — MOE, spatial, enhanced PUMS
phase3 — Estimates, flows, survey
integration — Requires real Census API key and network
spatial — Requires geopandas
```
120 changes: 111 additions & 9 deletions docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,22 +84,124 @@ print(type(la_poverty)) # <class 'geopandas.geodataframe.GeoDataFrame'>
4. **county** -- `"037"` is the FIPS code for Los Angeles County. Use `pypums.datasets.fips.lookup_fips(state="California", county="Los Angeles County")` to look up codes.
5. **geometry** -- When `True`, PyPUMS fetches TIGER/Line cartographic boundary shapefiles and merges them with the data. The result is a `GeoDataFrame` with a `geometry` column.

Now plot it:
Now plot it with [Altair](https://altair-viz.github.io/):

```python
la_poverty.plot(
column="estimate",
cmap="YlOrRd",
legend=True,
figsize=(12, 10),
edgecolor="0.8",
linewidth=0.3,
missing_kwds={"color": "lightgrey"},
import altair as alt

alt.Chart(la_poverty).mark_geoshape(
stroke="white", strokeWidth=0.3,
).encode(
color=alt.Color(
"estimate:Q",
scale=alt.Scale(scheme="yelloworangered"),
legend=alt.Legend(title="Below Poverty Level"),
),
tooltip=["NAME:N", alt.Tooltip("estimate:Q", format=",")],
).project("albersUsa").properties(
width=600, height=500,
title="Poverty by Census Tract, Los Angeles County",
)
```

The resulting map shows poverty counts by Census tract across Los Angeles County, with darker shades indicating higher counts.

!!! example "Interactive preview — state-level choropleth with ACS population data"
The chart below shows a state-level choropleth to demonstrate what
`geometry=True` output looks like when plotted with Altair. Your actual
tract-level map will have much finer geographic detail.

```vegalite
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"width": 600,
"height": 400,
"title": "ACS Total Population by State",
"data": {
"url": "https://cdn.jsdelivr.net/npm/vega-datasets@v2.7.0/data/us-10m.json",
"format": {"type": "topojson", "feature": "states"}
},
"transform": [
{
"lookup": "id",
"from": {
"data": {
"values": [
{"id": 1, "estimate": 5074296, "name": "Alabama"},
{"id": 2, "estimate": 733583, "name": "Alaska"},
{"id": 4, "estimate": 7359197, "name": "Arizona"},
{"id": 5, "estimate": 3045637, "name": "Arkansas"},
{"id": 6, "estimate": 39029342, "name": "California"},
{"id": 8, "estimate": 5839926, "name": "Colorado"},
{"id": 9, "estimate": 3626205, "name": "Connecticut"},
{"id": 10, "estimate": 1018396, "name": "Delaware"},
{"id": 11, "estimate": 671803, "name": "District of Columbia"},
{"id": 12, "estimate": 22244823, "name": "Florida"},
{"id": 13, "estimate": 10912876, "name": "Georgia"},
{"id": 15, "estimate": 1440196, "name": "Hawaii"},
{"id": 16, "estimate": 1939033, "name": "Idaho"},
{"id": 17, "estimate": 12582032, "name": "Illinois"},
{"id": 18, "estimate": 6833037, "name": "Indiana"},
{"id": 19, "estimate": 3200517, "name": "Iowa"},
{"id": 20, "estimate": 2937150, "name": "Kansas"},
{"id": 21, "estimate": 4512310, "name": "Kentucky"},
{"id": 22, "estimate": 4590241, "name": "Louisiana"},
{"id": 23, "estimate": 1385340, "name": "Maine"},
{"id": 24, "estimate": 6164660, "name": "Maryland"},
{"id": 25, "estimate": 6981974, "name": "Massachusetts"},
{"id": 26, "estimate": 10034113, "name": "Michigan"},
{"id": 27, "estimate": 5717184, "name": "Minnesota"},
{"id": 28, "estimate": 2940057, "name": "Mississippi"},
{"id": 29, "estimate": 6177957, "name": "Missouri"},
{"id": 30, "estimate": 1122867, "name": "Montana"},
{"id": 31, "estimate": 1967923, "name": "Nebraska"},
{"id": 32, "estimate": 3177772, "name": "Nevada"},
{"id": 33, "estimate": 1395231, "name": "New Hampshire"},
{"id": 34, "estimate": 9261699, "name": "New Jersey"},
{"id": 35, "estimate": 2113344, "name": "New Mexico"},
{"id": 36, "estimate": 19677151, "name": "New York"},
{"id": 37, "estimate": 10698973, "name": "North Carolina"},
{"id": 38, "estimate": 779261, "name": "North Dakota"},
{"id": 39, "estimate": 11756058, "name": "Ohio"},
{"id": 40, "estimate": 4019800, "name": "Oklahoma"},
{"id": 41, "estimate": 4240137, "name": "Oregon"},
{"id": 42, "estimate": 12972008, "name": "Pennsylvania"},
{"id": 44, "estimate": 1093734, "name": "Rhode Island"},
{"id": 45, "estimate": 5282634, "name": "South Carolina"},
{"id": 46, "estimate": 909824, "name": "South Dakota"},
{"id": 47, "estimate": 7051339, "name": "Tennessee"},
{"id": 48, "estimate": 30029572, "name": "Texas"},
{"id": 49, "estimate": 3380800, "name": "Utah"},
{"id": 50, "estimate": 647064, "name": "Vermont"},
{"id": 51, "estimate": 8642274, "name": "Virginia"},
{"id": 53, "estimate": 7785786, "name": "Washington"},
{"id": 54, "estimate": 1775156, "name": "West Virginia"},
{"id": 55, "estimate": 5892539, "name": "Wisconsin"},
{"id": 56, "estimate": 576851, "name": "Wyoming"}
]
},
"key": "id",
"fields": ["estimate", "name"]
}
}
],
"projection": {"type": "albersUsa"},
"mark": {"type": "geoshape", "stroke": "white", "strokeWidth": 0.5},
"encoding": {
"color": {
"field": "estimate",
"type": "quantitative",
"scale": {"scheme": "yelloworangered"},
"legend": {"title": "Population", "format": ","}
},
"tooltip": [
{"field": "name", "type": "nominal", "title": "State"},
{"field": "estimate", "type": "quantitative", "title": "Population", "format": ","}
]
}
}
```

!!! info "What is a TIGER/Line shapefile?"
The Census Bureau publishes free geographic boundary files called
TIGER/Line shapefiles. When you set `geometry=True`, PyPUMS automatically
Expand Down
9 changes: 7 additions & 2 deletions docs/guides/acs-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ The `output` parameter controls the shape of the returned DataFrame.
=== "Tidy (default)"

Each row is one geography-variable combination. This is ideal for
plotting with libraries like Altair, Plotly, or seaborn.
plotting with [Altair](https://altair-viz.github.io/).

```python
df_tidy = pypums.get_acs(
Expand Down Expand Up @@ -383,7 +383,12 @@ ca_counties_geo = pypums.get_acs(
print(type(ca_counties_geo))
# <class 'geopandas.geodataframe.GeoDataFrame'>

ca_counties_geo.plot(column="estimate", legend=True)
import altair as alt

alt.Chart(ca_counties_geo).mark_geoshape(stroke="white", strokeWidth=0.5).encode(
color=alt.Color("estimate:Q", legend=alt.Legend(title="Population")),
tooltip=["NAME:N", alt.Tooltip("estimate:Q", format=",")],
).project("albersUsa").properties(width=500, height=400)
```

!!! note "Optional dependency"
Expand Down
110 changes: 108 additions & 2 deletions docs/guides/decennial-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,108 @@ county_geo = pypums.get_decennial(
year=2020,
geometry=True,
)
county_geo.plot(column="value", legend=True)
import altair as alt

alt.Chart(county_geo).mark_geoshape(stroke="white", strokeWidth=0.5).encode(
color=alt.Color("value:Q", legend=alt.Legend(title="Population")),
tooltip=["NAME:N", alt.Tooltip("value:Q", format=",")],
).project("albersUsa").properties(width=500, height=400)
```

!!! example "Interactive preview — state-level 2020 Census population"
The code above fetches county-level data for Illinois. The chart below
uses state-level data to demonstrate what `geometry=True` + Altair
looks like at a broader geographic scale.

```vegalite
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"width": 500,
"height": 350,
"title": "2020 Census Population by State",
"data": {
"url": "https://cdn.jsdelivr.net/npm/vega-datasets@v2.7.0/data/us-10m.json",
"format": {"type": "topojson", "feature": "states"}
},
"transform": [
{
"lookup": "id",
"from": {
"data": {
"values": [
{"id": 1, "value": 5024279, "name": "Alabama"},
{"id": 2, "value": 733391, "name": "Alaska"},
{"id": 4, "value": 7151502, "name": "Arizona"},
{"id": 5, "value": 3011524, "name": "Arkansas"},
{"id": 6, "value": 39538223, "name": "California"},
{"id": 8, "value": 5773714, "name": "Colorado"},
{"id": 9, "value": 3605944, "name": "Connecticut"},
{"id": 10, "value": 989948, "name": "Delaware"},
{"id": 11, "value": 689545, "name": "District of Columbia"},
{"id": 12, "value": 21538187, "name": "Florida"},
{"id": 13, "value": 10711908, "name": "Georgia"},
{"id": 15, "value": 1455271, "name": "Hawaii"},
{"id": 16, "value": 1839106, "name": "Idaho"},
{"id": 17, "value": 12812508, "name": "Illinois"},
{"id": 18, "value": 6785528, "name": "Indiana"},
{"id": 19, "value": 3190369, "name": "Iowa"},
{"id": 20, "value": 2937880, "name": "Kansas"},
{"id": 21, "value": 4505836, "name": "Kentucky"},
{"id": 22, "value": 4657757, "name": "Louisiana"},
{"id": 23, "value": 1362359, "name": "Maine"},
{"id": 24, "value": 6177224, "name": "Maryland"},
{"id": 25, "value": 7029917, "name": "Massachusetts"},
{"id": 26, "value": 10077331, "name": "Michigan"},
{"id": 27, "value": 5706494, "name": "Minnesota"},
{"id": 28, "value": 2961279, "name": "Mississippi"},
{"id": 29, "value": 6154913, "name": "Missouri"},
{"id": 30, "value": 1084225, "name": "Montana"},
{"id": 31, "value": 1961504, "name": "Nebraska"},
{"id": 32, "value": 3104614, "name": "Nevada"},
{"id": 33, "value": 1377529, "name": "New Hampshire"},
{"id": 34, "value": 9288994, "name": "New Jersey"},
{"id": 35, "value": 2117522, "name": "New Mexico"},
{"id": 36, "value": 20201249, "name": "New York"},
{"id": 37, "value": 10439388, "name": "North Carolina"},
{"id": 38, "value": 779094, "name": "North Dakota"},
{"id": 39, "value": 11799448, "name": "Ohio"},
{"id": 40, "value": 3959353, "name": "Oklahoma"},
{"id": 41, "value": 4237256, "name": "Oregon"},
{"id": 42, "value": 13002700, "name": "Pennsylvania"},
{"id": 44, "value": 1097379, "name": "Rhode Island"},
{"id": 45, "value": 5118425, "name": "South Carolina"},
{"id": 46, "value": 886667, "name": "South Dakota"},
{"id": 47, "value": 6910840, "name": "Tennessee"},
{"id": 48, "value": 29145505, "name": "Texas"},
{"id": 49, "value": 3271616, "name": "Utah"},
{"id": 50, "value": 643077, "name": "Vermont"},
{"id": 51, "value": 8631393, "name": "Virginia"},
{"id": 53, "value": 7705281, "name": "Washington"},
{"id": 54, "value": 1793716, "name": "West Virginia"},
{"id": 55, "value": 5893718, "name": "Wisconsin"},
{"id": 56, "value": 576851, "name": "Wyoming"}
]
},
"key": "id",
"fields": ["value", "name"]
}
}
],
"projection": {"type": "albersUsa"},
"mark": {"type": "geoshape", "stroke": "white", "strokeWidth": 0.5},
"encoding": {
"color": {
"field": "value",
"type": "quantitative",
"scale": {"scheme": "blues"},
"legend": {"title": "Population", "format": ","}
},
"tooltip": [
{"field": "name", "type": "nominal", "title": "State"},
{"field": "value", "type": "quantitative", "title": "Population", "format": ","}
]
}
}
```

!!! note "Optional dependency"
Expand Down Expand Up @@ -381,7 +482,12 @@ pop_map = pypums.get_decennial(
year=2020,
geometry=True,
)
pop_map.plot(column="value", legend=True, cmap="YlOrRd")
import altair as alt

alt.Chart(pop_map).mark_geoshape(stroke="white", strokeWidth=0.3).encode(
color=alt.Color("value:Q", scale=alt.Scale(scheme="yelloworangered")),
tooltip=["NAME:N", alt.Tooltip("value:Q", format=",")],
).project("albersUsa").properties(width=500, height=400)
```

### Comparing 2010 and 2020
Expand Down
Loading
Loading