Ultra Fast Super Slick CI (#4755)

Attempting to improve our CI speeds as its recently been a pain point.

Major changes:

 - Use a github action to pull stable/nightly rust rather than building it each run
 - Shift test suite to `nexttest` https://github.com/nextest-rs/nextest for CI
 
 UPDATE:

So I've iterated on some changes, and although I think its still not optimal I think this is a good base to start from. Some extra things in this PR:
- Shifted where we pull rust from. We're now using this thing: https://github.com/moonrepo/setup-rust . It's got some interesting cache's built in, but was not seeing the gains that Jimmy managed to get. In either case tho, it can pull rust, cargofmt, clippy, cargo nexttest all in < 5s. So I think it's worthwhile. 
- I've grouped a few of the check-like tests into a single test called `code-test`. Although we were using github runners in parallel which may be faster, it just seems wasteful. There were like 4-5 tests, where we would pull lighthouse, compile it, then run an action, like clippy, cargo-audit or fmt. I've grouped these into a single action, so we only compile lighthouse once, then in each step we run the checks. This avoids compiling lighthouse like 5 times.
- Ive made doppelganger tests run on our local machines to avoid pulling foundry, building and making lcli which are all now baked into the images. 
- We have sccache and do not incremental compile lighthouse

Misc bonus things:
- Cargo update
- Fix web3 signer openssl keys which is required after a cargo update
- Use mock_instant in an LRU cache test to avoid non-deterministic test
- Remove race condition in building web3signer tests

There's still some things we could improve on. Such as downloading the EF tests every run and the web3-signer binary, but I've left these to be out of scope of this PR. I think the above are meaningful improvements.



Co-authored-by: Paul Hauner <paul@paulhauner.com>
Co-authored-by: realbigsean <seananderson33@gmail.com>
Co-authored-by: antondlr <anton@delaruelle.net>

.github/workflows/test-suite.yml

@@ -18,14 +18,14 @@ env:
   # Deny warnings in CI
   # Disable debug info (see https://github.com/sigp/lighthouse/issues/4005)
   RUSTFLAGS: "-D warnings -C debuginfo=0"
-  # The Nightly version used for cargo-udeps, might need updating from time to time.
-  PINNED_NIGHTLY: nightly-2023-04-16
   # Prevent Github API rate limiting.
   LIGHTHOUSE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   # Enable self-hosted runners for the sigp repo only.
   SELF_HOSTED_RUNNERS: ${{ github.repository == 'sigp/lighthouse' }}
   # Self-hosted runners need to reference a different host for `./watch` tests.
   WATCH_HOST: ${{ github.repository == 'sigp/lighthouse' && 'host.docker.internal' || 'localhost' }}
+  # Disable incremental compilation
+  CARGO_INCREMENTAL: 0
 jobs:
   target-branch-check:
     name: target-branch-check
@@ -34,145 +34,176 @@ jobs:
     steps:
         - name: Check that the pull request is not targeting the stable branch
           run: test ${{ github.base_ref }} != "stable"
-  extract-msrv:
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v3
-    - name: Extract Minimum Supported Rust Version (MSRV)
-      run: |
-        metadata=$(cargo metadata --no-deps --format-version 1)
-        msrv=$(echo $metadata | jq -r '.packages | map(select(.name == "lighthouse")) | .[0].rust_version')
-        echo "MSRV=$msrv" >> $GITHUB_OUTPUT
-      id: extract_msrv
-    outputs:
-      MSRV: ${{ steps.extract_msrv.outputs.MSRV }}
-  cargo-fmt:
-    name: cargo-fmt
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v3
-    - name: Get latest version of stable Rust
-      run: rustup update stable
-    - name: Check formatting with cargo fmt
-      run: make cargo-fmt
   release-tests-ubuntu:
     name: release-tests-ubuntu
     # Use self-hosted runners only on the sigp repo.
     runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest'  }}
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      if: env.SELF_HOSTED_RUNNERS == false
-      run: rustup update stable
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          bins: cargo-nextest
+      env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     - name: Install Foundry (anvil)
+      if: env.SELF_HOSTED_RUNNERS == 'false'
       uses: foundry-rs/foundry-toolchain@v1
       with:
         version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
     - name: Run tests in release
-      run: make test-release
+      run: make nextest-release
+    - name: Show cache stats
+      if: env.SELF_HOSTED_RUNNERS == 'true'
+      run: sccache --show-stats
   release-tests-windows:
     name: release-tests-windows
     runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "windows", "CI"]') || 'windows-2019'  }}
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      if: env.SELF_HOSTED_RUNNERS == false
-      run: rustup update stable
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          bins: cargo-nextest
+      env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     - name: Install Foundry (anvil)
+      if: env.SELF_HOSTED_RUNNERS == 'false'
       uses: foundry-rs/foundry-toolchain@v1
       with:
         version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
     - name: Install make
+      if: env.SELF_HOSTED_RUNNERS == 'false'
       run: choco install -y make
-    - uses: KyleMayes/install-llvm-action@v1
-      if: env.SELF_HOSTED_RUNNERS == false
-      with:
-        version: "15.0"
-        directory: ${{ runner.temp }}/llvm
+#    - uses: KyleMayes/install-llvm-action@v1
+#         if: env.SELF_HOSTED_RUNNERS == 'false'
+#      with:
+#        version: "15.0"
+#        directory: ${{ runner.temp }}/llvm
     - name: Set LIBCLANG_PATH
       run: echo "LIBCLANG_PATH=$((gcm clang).source -replace "clang.exe")" >> $env:GITHUB_ENV
     - name: Run tests in release
-      run: make test-release
+      run: make nextest-release
+    - name: Show cache stats
+      if: env.SELF_HOSTED_RUNNERS == 'true'
+      run: sccache --show-stats
   beacon-chain-tests:
     name: beacon-chain-tests
     # Use self-hosted runners only on the sigp repo.
     runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest'  }}
-    needs: cargo-fmt
+    env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      if: env.SELF_HOSTED_RUNNERS == false
-      run: rustup update stable
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          bins: cargo-nextest
     - name: Run beacon_chain tests for all known forks
       run: make test-beacon-chain
+    - name: Show cache stats
+      if: env.SELF_HOSTED_RUNNERS == 'true'
+      run: sccache --show-stats
   op-pool-tests:
     name: op-pool-tests
     runs-on: ubuntu-latest
-    needs: cargo-fmt
+    env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          bins: cargo-nextest
     - name: Run operation_pool tests for all known forks
       run: make test-op-pool
   slasher-tests:
     name: slasher-tests
     runs-on: ubuntu-latest
-    needs: cargo-fmt
+    env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          bins: cargo-nextest
     - name: Run slasher tests for all supported backends
       run: make test-slasher
   debug-tests-ubuntu:
     name: debug-tests-ubuntu
     # Use self-hosted runners only on the sigp repo.
     runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "large"]') || 'ubuntu-latest'  }}
-    needs: cargo-fmt
+    env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      if: env.SELF_HOSTED_RUNNERS == false
-      run: rustup update stable
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          bins: cargo-nextest
     - name: Install Foundry (anvil)
+      if: env.SELF_HOSTED_RUNNERS == 'false'
       uses: foundry-rs/foundry-toolchain@v1
       with:
         version: nightly-ca67d15f4abd46394b324c50e21e66f306a1162d
     - name: Run tests in debug
-      run: make test-debug
+      run: make nextest-debug
+    - name: Show cache stats
+      if: env.SELF_HOSTED_RUNNERS == 'true'
+      run: sccache --show-stats
   state-transition-vectors-ubuntu:
     name: state-transition-vectors-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
     - name: Run state_transition_vectors in release.
       run: make run-state-transition-tests
   ef-tests-ubuntu:
     name: ef-tests-ubuntu
     # Use self-hosted runners only on the sigp repo.
     runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "small"]') || 'ubuntu-latest'  }}
-    needs: cargo-fmt
+    env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      if: env.SELF_HOSTED_RUNNERS == false
-      run: rustup update stable
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          bins: cargo-nextest
     - name: Run consensus-spec-tests with blst, milagro and fake_crypto
-      run: make test-ef
+      run: make nextest-ef
+    - name: Show cache stats
+      if: env.SELF_HOSTED_RUNNERS == 'true'
+      run: sccache --show-stats
   dockerfile-ubuntu:
     name: dockerfile-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
-    - name: Get latest version of stable Rust
-      run: rustup update stable
     - name: Build the root Dockerfile
       run: docker build --build-arg FEATURES=portable -t lighthouse:local .
     - name: Test the built image
@@ -180,11 +211,13 @@ jobs:
   eth1-simulator-ubuntu:
     name: eth1-simulator-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
     - name: Install Foundry (anvil)
       uses: foundry-rs/foundry-toolchain@v1
       with:
@@ -194,11 +227,13 @@ jobs:
   merge-transition-ubuntu:
     name: merge-transition-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
     - name: Install Foundry (anvil)
       uses: foundry-rs/foundry-toolchain@v1
       with:
@@ -208,21 +243,25 @@ jobs:
   no-eth1-simulator-ubuntu:
     name: no-eth1-simulator-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
     - name: Run the beacon chain sim without an eth1 connection
       run: cargo run --release --bin simulator no-eth1-sim
   syncing-simulator-ubuntu:
     name: syncing-simulator-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
     - name: Install Foundry (anvil)
       uses: foundry-rs/foundry-toolchain@v1
       with:
@@ -231,21 +270,27 @@ jobs:
       run: cargo run --release --bin simulator syncing-sim
   doppelganger-protection-test:
     name: doppelganger-protection-test
-    runs-on: ubuntu-latest
-    needs: cargo-fmt
+    runs-on: ${{ github.repository == 'sigp/lighthouse' && fromJson('["self-hosted", "linux", "CI", "small"]') || 'ubuntu-latest'  }}
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
     - name: Install geth
+      if: env.SELF_HOSTED_RUNNERS == 'false'
       run: |
           sudo add-apt-repository -y ppa:ethereum/ethereum
           sudo apt-get update
           sudo apt-get install ethereum
-    - name: Install lighthouse and lcli
+    - name: Install lighthouse
       run: |
           make
-          make install-lcli
+    - name: Install lcli
+      if: env.SELF_HOSTED_RUNNERS == 'false'
+      run: make install-lcli
     - name: Run the doppelganger protection failure test script
       run: |
           cd scripts/tests
@@ -257,89 +302,71 @@ jobs:
   execution-engine-integration-ubuntu:
     name: execution-engine-integration-ubuntu
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-go@v3
-      with:
-        go-version: '1.20'
-    - uses: actions/setup-dotnet@v3
-      with:
-        dotnet-version: '6.0.201'
     - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          cache: false
+      env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     - name: Run exec engine integration tests in release
       run: make test-exec-engine
-  check-benchmarks:
-    name: check-benchmarks
+  check-code:
+    name: check-code
     runs-on: ubuntu-latest
-    needs: cargo-fmt
+    env:
+      CARGO_INCREMENTAL: 1
     steps:
     - uses: actions/checkout@v3
     - name: Get latest version of stable Rust
-      run: rustup update stable
-    - name: Typecheck benchmark code without running it
-      run: make check-benches
-  clippy:
-    name: clippy
-    runs-on: ubuntu-latest
-    needs: cargo-fmt
-    steps:
-    - uses: actions/checkout@v3
-    - name: Get latest version of stable Rust
-      run: rustup update stable
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: stable
+          cache-target: release
+          components: rustfmt,clippy
+          bins: cargo-audit
+    - name: Check formatting with cargo fmt
+      run: make cargo-fmt
     - name: Lint code for quality and style with Clippy
       run: make lint
     - name: Certify Cargo.lock freshness
       run: git diff --exit-code Cargo.lock
+    - name: Typecheck benchmark code without running it
+      run: make check-benches
+    - name: Validate state_processing feature arbitrary-fuzz
+      run:  make arbitrary-fuzz
+    - name: Run cargo audit
+      run: make audit-CI
+    - name: Run cargo vendor to make sure dependencies can be vendored for packaging, reproducibility and archival purpose
+      run:  CARGO_HOME=$(readlink -f $HOME) make vendor
   check-msrv:
     name: check-msrv
     runs-on: ubuntu-latest
-    needs: [cargo-fmt, extract-msrv]
     steps:
     - uses: actions/checkout@v3
-    - name: Install Rust @ MSRV (${{ needs.extract-msrv.outputs.MSRV }})
-      run: rustup override set ${{ needs.extract-msrv.outputs.MSRV }}
+    - name: Install Rust at Minimum Supported Rust Version (MSRV)
+      run: |
+        metadata=$(cargo metadata --no-deps --format-version 1)
+        msrv=$(echo $metadata | jq -r '.packages | map(select(.name == "lighthouse")) | .[0].rust_version')
+        rustup override set $msrv
     - name: Run cargo check
       run: cargo check --workspace
-  arbitrary-check:
-    name: arbitrary-check
-    runs-on: ubuntu-latest
-    needs: cargo-fmt
-    steps:
-    - uses: actions/checkout@v3
-    - name: Get latest version of stable Rust
-      run: rustup update stable
-    - name: Validate state_processing feature arbitrary-fuzz
-      run:  make arbitrary-fuzz
-  cargo-audit:
-    name: cargo-audit
-    runs-on: ubuntu-latest
-    needs: cargo-fmt
-    steps:
-    - uses: actions/checkout@v3
-    - name: Get latest version of stable Rust
-      run: rustup update stable
-    - name: Run cargo audit to identify known security vulnerabilities reported to the RustSec Advisory Database
-      run:  make audit
-  cargo-vendor:
-    name: cargo-vendor
-    runs-on: ubuntu-latest
-    needs: cargo-fmt
-    steps:
-    - uses: actions/checkout@v3
-    - name: Run cargo vendor to make sure dependencies can be vendored for packaging, reproducibility and archival purpose
-      run:  CARGO_HOME=$(readlink -f $HOME) make vendor
   cargo-udeps:
     name: cargo-udeps
     runs-on: ubuntu-latest
-    needs: cargo-fmt
     steps:
     - uses: actions/checkout@v3
-    - name: Install Rust (${{ env.PINNED_NIGHTLY }})
-      run: rustup toolchain install $PINNED_NIGHTLY
-    - name: Install cargo-udeps
-      run: cargo install cargo-udeps --locked --force
+    - name: Get latest version of nightly Rust
+      uses: moonrepo/setup-rust@v1
+      with:
+          channel: nightly
+          bins: cargo-udeps
+          cache: false
+      env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
     - name: Create Cargo config dir
       run: mkdir -p .cargo
     - name: Install custom Cargo config