repo-template-py

Repository template for Python projects under github.com/ianlewis.

This repository template is maintained for use in repositories under github.com/ianlewis. However, it can be used as a general purpose Python repository starter template.

Features

• Self-contained: Tools and linters are installed locally in the repository. Very few requirements.
• Development dependencies are tracked: Development dependencies are tracked using dependency files and lockfiles allowing them to be easily managed and upgraded; improving security and manageability.
• Reproducible: Because the repository is self-contained and development dependencies are tracked, linting and testing produce consistent results locally as well as on GitHub Actions.
• Tuned for OSS: Includes sane defaults for project documentation geared towards Open-Source projects and adhering to GitHub recommended community standards.
• Tuned for GitHub: Works well with GitHub checks and settings. Includes GitHub workflows for formatting and linting of base configuration files.
• Ideal for AI agents: Because the repository is self-contained, it is ideal for use with AI agents running in sandboxed environments. Tests and linters can be run easily by agents to check their work and get feedback on changes they have made.

Goals

Repository Quality

A set of formatters and linters are maintained to maintain repository code and configuration quality through pull request status checks.

Consistency & Reproducibility

Repositories created by this template should work as consistently as possible by minimizing issues due to conflicting installed package versions. Running commands and tools locally should have the same result between different local development machines and CI. Recommended language runtime versions are set via their respective ecosystem tooling.

This template strives to minimize outside dependencies on tools and configuration requiring only a minimal set of Unix userspace tools and language runtimes to work. Dependencies are downloaded and stored locally inside the project directory so they don't conflict with globally installed package versions.

Security

In general, dependencies for tools and GitHub Actions are pinned to improve overall project supply-chain security.

External dependencies on GitHub actions are limited to trusted actions with good security practices (e.g. official GitHub-owned actions) to minimize exposure to compromise via external repositories.

Versioning of formatting, linting, and other tool dependencies is done via lockfiles (e.g. uv.lock, package-lock.json). This is so that the versions can be maintained and updated via dependency automation tooling. This repository uses Mend Renovate because it allows more flexibility in configuration than Dependabot.

See also Recommended repository settings for more recommended security settings.

Requirements

This repository template supports the following operating systems and architectures.

• Linux x86-64 (AMD64).
• Linux aarch64 (ARM64).
• Darwin aarch64 (ARM64).

In general, dependencies on outside tools should be minimized in favor of including them as project-local dependencies.

1. Language runtimes

   The following language runtimes are required. It is recommended to use a tool that can manage multiple language runtime versions such as pyenv, nodenv, nvm, or asdf. This repository includes .node-version and .python-version files to specify the language runtime versions to use for maximum compatibility with these tools.

   • Node.js: Node.js is required to run some linters and formatters.
   • Python: Python is required to run some linters and formatters.

2. System tools

   The following tools need to be installed:

   • git: For repository management.
   • awk, basename, bash, dirname, grep, head, mktemp, rm, sha256sum, uname: Standard Unix tools (GNU version).
   • GNU make: For running commands.
   • curl, tar, gzip: For extracting archives.

   On macOS you can install the required packages with Homebrew.

   brew install \
       make \
       grep \
       gawk \
       coreutils \
       libyaml \
       xz

The following tools are automatically installed locally to the project and do not need to be pre-installed:

• actionlint: For linting GitHub Actions workflows (installed by Aqua in .aqua).
• checkmake: For linting Makefile (installed by Aqua in .aqua).
• commitlint: For checking commit messages (installed by local node_modules).
• jq: For parsing output of some linters (installed by Aqua in .aqua).
• markdownlint: For linting Markdown (installed in local node_modules).
• mbrukman/autogen: For adding license headers (vendored in third_party).
• prettier: For formatting Markdown and YAML files (installed in local node_modules).
• ruff: For formatting and linting Python code (installed by Aqua in .aqua).
• shellcheck: For linting shell code in GitHub Actions workflows (installed by Aqua in .aqua).
• textlint: For spelling checks (installed in local node_modules).
• todos: For checking for outstanding TODOs in code (installed by Aqua in .aqua).
• yamllint: For linting YAML files (installed in local Python virtualenv .venv).
• zizmor: For linting GitHub Actions workflows (installed in local Python virtualenv .venv).

Usage

The repository is organized to be as self-contained as possible. Commands are implemented in the project Makefile.

Makefile

The Makefile is used for running commands, managing files, and maintaining code quality. It includes a default help target that prints all make targets and their descriptions grouped by function.

$ make
repo-template-py Makefile
Usage: make [COMMAND]

  help                      Print all Makefile targets (this message).
Build
  all                       Run all tests and build a release package.
  package                   Create a release package.
Testing
  test                      Run all linters and tests.
  unit-test                 Run unit tests.
Formatting
  format                    Format all files
  json-format               Format JSON files.
  license-headers           Update license headers.
  md-format                 Format Markdown files.
  py-format                 Format Python files.
  yaml-format               Format YAML files.
Linting
  lint                      Run all linters.
  actionlint                Runs the actionlint linter.
  checkmake                 Runs the checkmake linter.
  commitlint                Run commitlint linter.
  fixme                     Check for outstanding FIXMEs.
  format-check              Check that files are properly formatted.
  markdownlint              Runs the markdownlint linter.
  mypy                      Runs the mypy type checker.
  renovate-config-validator Validate Renovate configuration.
  ruff                      Runs the ruff linter.
  textlint                  Runs the textlint linter.
  yamllint                  Runs the yamllint linter.
  zizmor                    Runs the zizmor linter.
Maintenance
  update-lockfiles          Update lockfiles.
  todos                     Print outstanding TODOs.
  clean                     Delete temporary files.

Formatting and Linting

Some Makefile targets for basic formatters and linters are included along with GitHub Actions pre-submits. Where possible, pre-submits use Makefile targets and those targets execute with the same settings as they do when run locally. This is to give a consistent experience when attempting to reproduce pre-submit errors.

Versioning of formatting, linting, and other tools are managed as tool dependencies so they can be more easily maintained.

Makefile targets and linter/formatter configuration are designed to respect .gitignore and not cross git submodule boundaries. However, you will need to add files using git add for new files before they are picked up.

Makefile targets for linters will also produce human-readable output by default, but will produce errors as GitHub Actions workflow commands so they can be easily interpreted when run in Pull-Request status checks.

License Headers

The license-headers make target will add license headers to files that are missing it with the Copyright holder set to the current value of git config user.name.

Files are checked for the existence license headers in status checks.

Updating Dependencies

You can update dependencies by updating the appropriate dependency file (package.json, .aqua.yaml, etc.) and running make update-lockfiles. This will update the lockfiles with the appropriate versions and checksums so that they can be installed consistently.

Mend Renovate can be used to automatically update dependencies and lockfiles. The autofix.ci GitHub App can also be used to automatically update lockfiles on pull requests created by Renovate.

Testing

Running make test will run all tests including linters. This target is meant to run all tests necessary to validate code for deployment including unit tests, integration tests, and end-to-end tests.

Running make unit-test will run unit tests only. This target will generate coverage using coverage.py and output the results to .coverage.

Conventional Commits

This repository template uses Conventional Commits to standardize commit message formatting. Conventional commits can help to communicate the nature of changes at a glance, and give hints on backwards compatibility.

While you may use conventional commits to automate releases, it is not recommended to use commit messages to automatically determine the next release version, or to auto-generate user-facing documentation such as the CHANGELOG.md or release notes. These should be written for an end-user audience, be human readable, and include additional relevant information and context.

This repository deviates slightly from the conventional commits specification by requiring that all commits include a scope. Scope is project specific and could refer to a component, module, or section of the codebase.

This post explains some of the rationale:

• Stop Using Conventional Commits - Sumner Evans

Keeping Downstream Repositories in Sync

You can optionally keep repositories created with the template in sync with changes to the template. Because repositories created from GitHub templates are not forks, it is recommended to perform a squash merge so the template sync lands as a single commit on your commit history.

# One time step: Add the repository template as a remote.
git remote add repo-template git@github.com:ianlewis/repo-template.git

# Fetch the latest version of the repo-template.
git fetch repo-template main

# Create a new squash merge commit.
git merge --no-edit --signoff --squash --allow-unrelated-histories repo-template/main

Repository Creation Checklist

When creating a new repository from this template, the following checklist is recommended to ensure the repository is set up correctly.

Update Configuration Files

Files that should be updated include a TODO comment to indicate what changes should made. You can run make todos to list all TODOs in the repository.

$ make todos
.github/workflows/pre-submit.units.yml:113:# TODO: Remove the next line for private repositories with GitHub Advanced Security.
.github/workflows/schedule.scorecard.yml:80:# TODO: Remove the next line for private repositories with GitHub Advanced Security.
CODEOWNERS:1:# TODO: Update CODEOWNERS
CODE_OF_CONDUCT.md:61:<!-- TODO: update Code of Conduct contact email -->
README.md:3:<!-- TODO: update badge urls -->
README.md:7:<!-- TODO: Update README contents. -->

Recommended Repository Settings

The following repository settings are recommended in conjunction with this repository template.

General / Pull Requests

1. Allow merge commits: Enabling merge commits is recommended to preserve the original commit history of a pull request. This makes it easier to retain the original commit messages and signatures of contributors. Disallowing squash merges and rebase merges is recommended.
2. Require contributors to sign off on web-based commits: This is recommended to ensure that all commits made through the GitHub web interface are signed off by the author with a Developer Certificate of Origin (DCO).

GitHub Apps

1. Enable DCO: Enable the DCO GitHub App to ensure that all commits are signed off by the author with a DCO. This is a lightweight alternative to a Contributor License Agreement (CLA) for contributors to certify that they wrote or otherwise have the right to submit the code they are contributing to the project. If you have a CLA you can omit this step.
2. Enable Mend Renovate: Enable Renovate to automatically update dependencies and lockfiles. This is recommended to ensure that dependencies are kept up to date and security vulnerabilities are addressed in a timely manner.
3. Enable autofix.ci: Enable autofix.ci to automatically update lockfiles in pull requests. This is recommended to avoid having to manually update lockfiles on Renovate PRs.

Rules / Rulesets

A ruleset should be created for the default branch with branch protection rules that follow the recommendations from OpenSSF Scorecard to achieve the highest Tier and score as possible.

Branch Rules

• Require a pull request before merging: This is recommended to ensure that all changes to the default branch are reviewed and the code passes status checks. This helps maintain code quality and security.

• Require status checks to pass: This is recommended to ensure that all pull requests pass the required checks before they can be merged. This helps maintain code quality and prevents broken code from being merged into the default branch.

  Status checks that are required: The following checks should be marked as required:

  • DCO
  • actionlint / actionlint
  • checkmake / checkmake
  • commitlint / commitlint
  • format-check / format-check
  • fixme / fixme
  • markdownlint / markdownlint
  • mypy / mypy
  • renovate-config-validator / renovate-config-validator
  • ruff / ruff
  • textlint / textlint
  • unit-test / unit-test
  • yamllint / yamllint
  • zizmor / zizmor

Block Force Pushes

• Block force pushes: This is recommended to ensure that all changes to the default branch are made via pull requests.

Require Code Scanning Results

The following tools should be added to the required code scanning results.

• CodeQL
• zizmor

Advanced Security

1. Private vulnerability reporting: Enable private vulnerability reporting as mentioned in SECURITY.md.
2. Dependency Graph: Enable the dependency graph and automatic dependency submission. Renovate relies on dependency graph for its vulnerability alerts feature.
3. Dependabot Alerts: Enable Dependabot alerts. Renovate relies on Dependabot alerts for its vulnerability alerts feature.

Code Scanning

1. CodeQL analysis: Make sure "GitHub Actions (Public Preview)" is enabled in languages.
2. Protection rules:
   • Security alert severity level: Errors and warnings
   • Standard alert severity level: Errors and warnings
3. Secret protection: Get alerts when secrets are detected in the repository.
4. Push protection: Block pushing commits with secrets in them.

Secrets and variables / Actions

Add the repository to Codecov for code coverage. View the Codecov Quick Start Guide for more information.

• CODECOV_TOKEN: add the repository upload token as a repository secret.

Copilot / Cloud Agent

If you use GitHub Copilot in your repository, it is recommended to enable the following settings:

1. Custom Allowlist: Add the following domains to the Copilot custom allowlist to allow Copilot to access them in agent sessions.
   • sigstore.dev

Contributing

PRs may be accepted to this template. This template uses the same CONTRIBUTING.md for contributor documentation as for new repositories created from the template.