2f807e21be
This PR adds the ability to download [nightly reference tests from the consensus-specs repo](https://github.com/ethereum/consensus-specs/actions/workflows/generate_vectors.yml). This will be used by spec maintainers to ensure that there are no unexpected test failures prior to new releases. Also, we will keep track of test compliance with [this website](https://jtraglia.github.io/nyx/); eventually this will be integrated into Hive. * A new script (`download_test_vectors.sh`) is added to handle downloads. * The logic for downloading GitHub artifacts is a bit complex. * Rename the variables which store test versions: * `TESTS_TAG` to `CONSENSUS_SPECS_TEST_VERSION`. * `BLS_TEST_TAG` to `BLS_TEST_VERSION`, for consistency. * Delete tarballs after extracting them. * I see no need to keep these; they just use extra disk. * Consolidate `clean` rules into a single rule. * Do `clean` prior to downloading/extracting tests. * Remove `CURL` variable with GitHub token; don't need it for downloading releases. * Do `mkdir -p` when creating directories. * Probably more small stuff...
Consensus Specification Tests
This crate parses and executes the test vectors at ethereum/consensus-spec-tests.
Functionality is achieved only via the $ cargo test --features ef_tests command.
Running the Tests
Because the test vectors are very large, we do not download or run them by default. To download them, run (in this directory):
$ make
Note: this may download hundreds of MB of compressed archives from the ethereum/consensus-spec-tests, which may expand into several GB of files.
If successful, you should now have the extracted tests in ./consensus-spec-tests.
Run them with:
$ cargo test --features ef_tests
The tests won't run without the ef_tests feature enabled (this is to ensure that a top-level
cargo test --all won't fail on missing files).
The following is sometimes necessary to avoid stack overflow issues when running on MacOS:
$ export RUST_MIN_STACK=8388608
When debugging failing tests, it's often useful to disable parallization and output suppression:
$ cargo test --features ef_tests,disable_rayon -- --nocapture
Saving Space
When you download the tests, the downloaded archives will be kept in addition to the extracted files. You have several options for saving space:
- Delete the archives (
make clean-archives), and keep the extracted files. Suitable for everyday use, just don't re-runmakeor it will redownload the archives. - Delete the extracted files (
make clean-test-files), and keep the archives. Suitable for CI, or temporarily saving space. If you re-runmakeit will extract the archives rather than redownloading them. - Delete everything (
make clean). Good for updating to a new version, or if you no longer wish to run the EF tests.