Add CLI integration tests, smoke wrappers, and CI workflow#39
Closed
tbdye wants to merge 2 commits into
Closed
Conversation
tbdye
force-pushed
the
phase5/cli-tests-ci
branch
2 times, most recently
from
ee1fa80 to
723a31d
Compare
tbdye
force-pushed
the
phase5/cli-tests-ci
branch
from
723a31d to
11f6728
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Builds on #38 to complete the test story. CLI subprocess tests validate the JSON output schema of every
amifusesubcommand. Smoke test wrappers bring the existing tools/ scripts under pytest for unified discovery. A GitHub Actions CI workflow runs unit tests across the full matrix and integration tests on supported platforms.Details
CLI tests (
test_cli_commands.py, 31 tests): Runpython -m amifuse <cmd>as a subprocess, parse the JSON envelope, and validate structure and known values for inspect, doctor, ls, verify, hash, read, and write subcommands. A_parse_json_stdouthelper strips vamos debug noise that can precede the JSON object.Smoke test wrappers (
test_matrix_smoke.py,test_readme_smoke.py,test_image_format_smoke.py): Thin subprocess launchers marked@pytest.mark.smokethat run the existing tools/ scripts under pytest. Auto-skip when external fixtures aren't present.CI workflow (
.github/workflows/ci.yml): Two-stage pipeline — unit tests first (3 OS × 3 Python = 9 jobs), then integration tests (2 jobs on Ubuntu and macOS, depends on unit pass).Windows integration tests are excluded from CI. The machine68k C extension segfaults on Windows due to two upstream bugs: an opcode table over-read (cnvogelg/machine68k#8) and a JMP/CAS opcode collision under MSVC (cnvogelg/machine68k#9). The integration conftest probes
machine68k.CPU(1)in a subprocess to survive the crash gracefully, but until those PRs merge, all integration tests would be silently skipped on Windows — so we exclude the platform from CI rather than report false greens. Windows integration tests have been validated locally against a patched machine68k build. Re-enablewindows-latestin the integration matrix once machine68k merges #8 and #9.TESTING.md: Updated with the layered test architecture (unit → integration → smoke), pytest usage, and CI documentation.
This PR is stacked on #38 — it should be rebased onto main after that PR merges.
Testing
31 CLI integration tests, 3 smoke test wrappers. All CLI tests pass on Windows locally with Python 3.12+ and a patched machine68k. Smoke wrappers auto-skip gracefully when external test fixtures aren't downloaded. CI workflow validated: 9 unit jobs (3 OS × 3 Python) + 2 integration jobs (Ubuntu, macOS) all green.