Skip to content

fix: make block_number optional + allow non-interactive pull all#16

Open
greatmerlin wants to merge 2 commits into
coko7:mainfrom
greatmerlin:fix/optional-block-number
Open

fix: make block_number optional + allow non-interactive pull all#16
greatmerlin wants to merge 2 commits into
coko7:mainfrom
greatmerlin:fix/optional-block-number

Conversation

@greatmerlin

@greatmerlin greatmerlin commented Jun 7, 2026

Copy link
Copy Markdown

Summary

Two changes, both prompted by vega pull all aborting on the newest cards.

1. fix: don't abort the whole scrape when a card has no Standard block

The newest cards on the official site (e.g. OP16-063 Kuzan) have no Standard block assigned yet, so the dd>div.backCol>div.col2>div.block element is empty. fetch_block_number treated this as fatal:

ERROR vega] failed to scrape data about card `OP16-063`: card.block_number is empty!

…which aborts the entire pull all run partway through.

This changes Card.block_number from i32 to Option<i32>, and fetch_block_number now returns Ok(None) when the block node is absent or contains no digits, instead of bail!-ing. Genuine parse errors (non-empty, non-numeric) still error.

Note: this makes block_number nullable in the serialized output — such cards now emit "block_number": null instead of crashing the run.

2. feat: non-interactive pull all via -o and --with-images

pull all was always interactive (inquire prompts for language, output dir, images), so it couldn't be scripted or run in CI. Now, when -o/--output is supplied, pull all skips the prompts and takes the language from -l/--language and the image choice from a new -a/--with-images flag (mirroring pull cards). With no -o, the interactive flow is unchanged.

# scriptable now:
vega pull -l english -o ./data all        # cards only
vega pull -l english -o ./data all -a     # cards + images

Testing

  • cargo build --release is clean.
  • vega pull -l english -o data all -a scrapes all 53 packs + 4571 images with no crash; OP16-063 is captured with block_number: null.

greatmerlin and others added 2 commits June 7, 2026 21:16
The newest cards on the official site (e.g. OP16-063 Kuzan) have no Standard
block assigned yet, so the `div.block` element is empty. `fetch_block_number`
treated this as fatal (`bail!("card.block_number is empty!")`), which aborted
the entire `pull all` run mid-way.

Change `Card.block_number` to `Option<i32>` and have `fetch_block_number`
return `Ok(None)` when the block node is absent or contains no digits, rather
than bailing. Genuine parse errors (non-empty, non-numeric) still error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`pull all` was always interactive (inquire prompts for language, output dir,
and images), which made it impossible to script or run in CI. Now, when an
output directory is supplied with `-o/--output`, `pull all` skips every prompt
and takes the language from `-l/--language` and the image choice from a new
`-a/--with-images` flag (mirroring `pull cards`). With no `-o`, the original
interactive prompts are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coko7

coko7 commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Hi @greatmerlin,
Thank you for your contribution.

It seems your fork is not rebased to the latest upstream commit. Part of your PR was already fixed by #10.
Please rebase first, I will gladly review your PR afterwards :)

@coko7 coko7 self-requested a review June 8, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants