feat: add --order option to run tests in random order#6144
Open
kirthi-b wants to merge 1 commit into
Open
Conversation
|
|
Member
|
Looks OK, pretty big and low-priority, but I'll add it to my backlog :) |
Member
|
FYI @JoshuaKGoldberg the backing issue has 80+ likes! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist
status: accepting prsOverview
Implements the
--orderoption from #902, following the design approved in#902 (comment) and the
option shape from the issue body:
--order default,--order randomand--order random:<seed>.How it works:
parent suite, per 🚀 Feature: --order option for random test order #902 (comment).
The hierarchy is preserved, so
before/after/beforeEach/afterEachrunexactly as often as in a normal run.
Mocha: test order randomized with seed 42; use --order random:42 to reproduce. The seed goes to STDERR, somachine-readable reporters on STDOUT (json, xunit) are unaffected. This was
the main open question on the old reference PR feat: randomization support [WIP REFERENCE - DO NOT MERGE] #2301.
Runner#run's prepare step, so it also works with--delayand inside parallel workers. In parallel mode the main processbakes the generated seed into the options before they are serialized, so all
workers shuffle from the same seed and the seed is printed once.
--sortand--order randomerror out as mutually exclusive, same as theexisting
--paralleland--sortpairing.mocha.order('random:42')and theorderconstructor option.Includes unit tests, integration tests (fixed seed order, reproduce-from-
reported-seed round trip, error cases) and docs for the new flag.
Notes for reviewers:
randomizationbranch, but thatbranch no longer exists on the remote, so this targets
main.inline seeded PRNG (mulberry32) to avoid adding a runtime dependency;
identical seeds give identical orders on any platform. Happy to switch to
seedrandom if you prefer the dependency.
would rather surface it through reporters or a runner event, I can rework
that.
randomanddefaultare implemented; therandom-suitesvariantfloated mid-thread was left out to keep scope to the approved design.
Closes #902