Use async code when interacting with EL (#3244)

## Overview

This rather extensive PR achieves two primary goals:

1. Uses the finalized/justified checkpoints of fork choice (FC), rather than that of the head state.
2. Refactors fork choice, block production and block processing to `async` functions.

Additionally, it achieves:

- Concurrent forkchoice updates to the EL and cache pruning after a new head is selected.
- Concurrent "block packing" (attestations, etc) and execution payload retrieval during block production.
- Concurrent per-block-processing and execution payload verification during block processing.
- The `Arc`-ification of `SignedBeaconBlock` during block processing (it's never mutated, so why not?):
    - I had to do this to deal with sending blocks into spawned tasks.
    - Previously we were cloning the beacon block at least 2 times during each block processing, these clones are either removed or turned into cheaper `Arc` clones.
    - We were also `Box`-ing and un-`Box`-ing beacon blocks as they moved throughout the networking crate. This is not a big deal, but it's nice to avoid shifting things between the stack and heap.
    - Avoids cloning *all the blocks* in *every chain segment* during sync.
    - It also has the potential to clean up our code where we need to pass an *owned* block around so we can send it back in the case of an error (I didn't do much of this, my PR is already big enough 😅)
- The `BeaconChain::HeadSafetyStatus` struct was removed. It was an old relic from prior merge specs.

For motivation for this change, see https://github.com/sigp/lighthouse/pull/3244#issuecomment-1160963273

## Changes to `canonical_head` and `fork_choice`

Previously, the `BeaconChain` had two separate fields:

```
canonical_head: RwLock<Snapshot>,
fork_choice: RwLock<BeaconForkChoice>
```

Now, we have grouped these values under a single struct:

```
canonical_head: CanonicalHead {
  cached_head: RwLock<Arc<Snapshot>>,
  fork_choice: RwLock<BeaconForkChoice>
} 
```

Apart from ergonomics, the only *actual* change here is wrapping the canonical head snapshot in an `Arc`. This means that we no longer need to hold the `cached_head` (`canonical_head`, in old terms) lock when we want to pull some values from it. This was done to avoid deadlock risks by preventing functions from acquiring (and holding) the `cached_head` and `fork_choice` locks simultaneously.

## Breaking Changes

### The `state` (root) field in the `finalized_checkpoint` SSE event

Consider the scenario where epoch `n` is just finalized, but `start_slot(n)` is skipped. There are two state roots we might in the `finalized_checkpoint` SSE event:

1. The state root of the finalized block, which is `get_block(finalized_checkpoint.root).state_root`.
4. The state root at slot of `start_slot(n)`, which would be the state from (1), but "skipped forward" through any skip slots.

Previously, Lighthouse would choose (2). However, we can see that when [Teku generates that event](de2b2801c8/data/beaconrestapi/src/main/java/tech/pegasys/teku/beaconrestapi/handlers/v1/events/EventSubscriptionManager.java (L171-L182)) it uses [`getStateRootFromBlockRoot`](de2b2801c8/data/provider/src/main/java/tech/pegasys/teku/api/ChainDataProvider.java (L336-L341)) which uses (1).

I have switched Lighthouse from (2) to (1). I think it's a somewhat arbitrary choice between the two, where (1) is easier to compute and is consistent with Teku.

## Notes for Reviewers

I've renamed `BeaconChain::fork_choice` to `BeaconChain::recompute_head`. Doing this helped ensure I broke all previous uses of fork choice and I also find it more descriptive. It describes an action and can't be confused with trying to get a reference to the `ForkChoice` struct.

I've changed the ordering of SSE events when a block is received. It used to be `[block, finalized, head]` and now it's `[block, head, finalized]`. It was easier this way and I don't think we were making any promises about SSE event ordering so it's not "breaking".

I've made it so fork choice will run when it's first constructed. I did this because I wanted to have a cached version of the last call to `get_head`. Ensuring `get_head` has been run *at least once* means that the cached values doesn't need to wrapped in an `Option`. This was fairly simple, it just involved passing a `slot` to the constructor so it knows *when* it's being run. When loading a fork choice from the store and a slot clock isn't handy I've just used the `slot` that was saved in the `fork_choice_store`. That seems like it would be a faithful representation of the slot when we saved it.

I added the `genesis_time: u64` to the `BeaconChain`. It's small, constant and nice to have around.

Since we're using FC for the fin/just checkpoints, we no longer get the `0x00..00` roots at genesis. You can see I had to remove a work-around in `ef-tests` here: b56be3bc2. I can't find any reason why this would be an issue, if anything I think it'll be better since the genesis-alias has caught us out a few times (0x00..00 isn't actually a real root). Edit: I did find a case where the `network` expected the 0x00..00 alias and patched it here: 3f26ac3e2.

You'll notice a lot of changes in tests. Generally, tests should be functionally equivalent. Here are the things creating the most diff-noise in tests:
- Changing tests to be `tokio::async` tests.
- Adding `.await` to fork choice, block processing and block production functions.
- Refactor of the `canonical_head` "API" provided by the `BeaconChain`. E.g., `chain.canonical_head.cached_head()` instead of `chain.canonical_head.read()`.
- Wrapping `SignedBeaconBlock` in an `Arc`.
- In the `beacon_chain/tests/block_verification`, we can't use the `lazy_static` `CHAIN_SEGMENT` variable anymore since it's generated with an async function. We just generate it in each test, not so efficient but hopefully insignificant.

I had to disable `rayon` concurrent tests in the `fork_choice` tests. This is because the use of `rayon` and `block_on` was causing a panic.

Co-authored-by: Mac L <mjladson@pm.me>

consensus/state_processing/src/per_block_processing/tests.rs

@@ -27,7 +27,7 @@ lazy_static! {
     static ref KEYPAIRS: Vec<Keypair> = generate_deterministic_keypairs(MAX_VALIDATOR_COUNT);
 }
 
-fn get_harness<E: EthSpec>(
+async fn get_harness<E: EthSpec>(
     epoch_offset: u64,
     num_validators: usize,
 ) -> BeaconChainHarness<EphemeralHarnessType<E>> {
@@ -41,27 +41,31 @@ fn get_harness<E: EthSpec>(
         .build();
     let state = harness.get_current_state();
     if last_slot_of_epoch > Slot::new(0) {
-        harness.add_attested_blocks_at_slots(
-            state,
-            Hash256::zero(),
-            (1..last_slot_of_epoch.as_u64())
-                .map(Slot::new)
-                .collect::<Vec<_>>()
-                .as_slice(),
-            (0..num_validators).collect::<Vec<_>>().as_slice(),
-        );
+        harness
+            .add_attested_blocks_at_slots(
+                state,
+                Hash256::zero(),
+                (1..last_slot_of_epoch.as_u64())
+                    .map(Slot::new)
+                    .collect::<Vec<_>>()
+                    .as_slice(),
+                (0..num_validators).collect::<Vec<_>>().as_slice(),
+            )
+            .await;
     }
     harness
 }
 
-#[test]
-fn valid_block_ok() {
+#[tokio::test]
+async fn valid_block_ok() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let state = harness.get_current_state();
 
     let slot = state.slot();
-    let (block, mut state) = harness.make_block_return_pre_state(state, slot + Slot::new(1));
+    let (block, mut state) = harness
+        .make_block_return_pre_state(state, slot + Slot::new(1))
+        .await;
 
     let result = per_block_processing(
         &mut state,
@@ -75,15 +79,15 @@ fn valid_block_ok() {
     assert!(result.is_ok());
 }
 
-#[test]
-fn invalid_block_header_state_slot() {
+#[tokio::test]
+async fn invalid_block_header_state_slot() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let state = harness.get_current_state();
     let slot = state.slot() + Slot::new(1);
 
-    let (signed_block, mut state) = harness.make_block_return_pre_state(state, slot);
+    let (signed_block, mut state) = harness.make_block_return_pre_state(state, slot).await;
     let (mut block, signature) = signed_block.deconstruct();
     *block.slot_mut() = slot + Slot::new(1);
 
@@ -104,15 +108,17 @@ fn invalid_block_header_state_slot() {
     );
 }
 
-#[test]
-fn invalid_parent_block_root() {
+#[tokio::test]
+async fn invalid_parent_block_root() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let state = harness.get_current_state();
     let slot = state.slot();
 
-    let (signed_block, mut state) = harness.make_block_return_pre_state(state, slot + Slot::new(1));
+    let (signed_block, mut state) = harness
+        .make_block_return_pre_state(state, slot + Slot::new(1))
+        .await;
     let (mut block, signature) = signed_block.deconstruct();
     *block.parent_root_mut() = Hash256::from([0xAA; 32]);
 
@@ -136,14 +142,16 @@ fn invalid_parent_block_root() {
     );
 }
 
-#[test]
-fn invalid_block_signature() {
+#[tokio::test]
+async fn invalid_block_signature() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let state = harness.get_current_state();
     let slot = state.slot();
-    let (signed_block, mut state) = harness.make_block_return_pre_state(state, slot + Slot::new(1));
+    let (signed_block, mut state) = harness
+        .make_block_return_pre_state(state, slot + Slot::new(1))
+        .await;
     let (block, _) = signed_block.deconstruct();
 
     let result = per_block_processing(
@@ -164,17 +172,19 @@ fn invalid_block_signature() {
     );
 }
 
-#[test]
-fn invalid_randao_reveal_signature() {
+#[tokio::test]
+async fn invalid_randao_reveal_signature() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let state = harness.get_current_state();
     let slot = state.slot();
 
-    let (signed_block, mut state) = harness.make_block_with_modifier(state, slot + 1, |block| {
-        *block.body_mut().randao_reveal_mut() = Signature::empty();
-    });
+    let (signed_block, mut state) = harness
+        .make_block_with_modifier(state, slot + 1, |block| {
+            *block.body_mut().randao_reveal_mut() = Signature::empty();
+        })
+        .await;
 
     let result = per_block_processing(
         &mut state,
@@ -189,16 +199,22 @@ fn invalid_randao_reveal_signature() {
     assert_eq!(result, Err(BlockProcessingError::RandaoSignatureInvalid));
 }
 
-#[test]
-fn valid_4_deposits() {
+#[tokio::test]
+async fn valid_4_deposits() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut state = harness.get_current_state();
 
     let (deposits, state) = harness.make_deposits(&mut state, 4, None, None);
     let deposits = VariableList::from(deposits);
 
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     *head_block.to_mut().body_mut().deposits_mut() = deposits;
 
     let result = process_operations::process_deposits(state, head_block.body().deposits(), &spec);
@@ -207,16 +223,22 @@ fn valid_4_deposits() {
     assert_eq!(result, Ok(()));
 }
 
-#[test]
-fn invalid_deposit_deposit_count_too_big() {
+#[tokio::test]
+async fn invalid_deposit_deposit_count_too_big() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut state = harness.get_current_state();
 
     let (deposits, state) = harness.make_deposits(&mut state, 1, None, None);
     let deposits = VariableList::from(deposits);
 
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     *head_block.to_mut().body_mut().deposits_mut() = deposits;
 
     let big_deposit_count = NUM_DEPOSITS + 1;
@@ -233,16 +255,22 @@ fn invalid_deposit_deposit_count_too_big() {
     );
 }
 
-#[test]
-fn invalid_deposit_count_too_small() {
+#[tokio::test]
+async fn invalid_deposit_count_too_small() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut state = harness.get_current_state();
 
     let (deposits, state) = harness.make_deposits(&mut state, 1, None, None);
     let deposits = VariableList::from(deposits);
 
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     *head_block.to_mut().body_mut().deposits_mut() = deposits;
 
     let small_deposit_count = NUM_DEPOSITS - 1;
@@ -259,16 +287,22 @@ fn invalid_deposit_count_too_small() {
     );
 }
 
-#[test]
-fn invalid_deposit_bad_merkle_proof() {
+#[tokio::test]
+async fn invalid_deposit_bad_merkle_proof() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut state = harness.get_current_state();
 
     let (deposits, state) = harness.make_deposits(&mut state, 1, None, None);
     let deposits = VariableList::from(deposits);
 
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     *head_block.to_mut().body_mut().deposits_mut() = deposits;
     let bad_index = state.eth1_deposit_index() as usize;
 
@@ -287,17 +321,23 @@ fn invalid_deposit_bad_merkle_proof() {
     );
 }
 
-#[test]
-fn invalid_deposit_wrong_sig() {
+#[tokio::test]
+async fn invalid_deposit_wrong_sig() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut state = harness.get_current_state();
 
     let (deposits, state) =
         harness.make_deposits(&mut state, 1, None, Some(SignatureBytes::empty()));
     let deposits = VariableList::from(deposits);
 
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     *head_block.to_mut().body_mut().deposits_mut() = deposits;
 
     let result = process_operations::process_deposits(state, head_block.body().deposits(), &spec);
@@ -305,17 +345,23 @@ fn invalid_deposit_wrong_sig() {
     assert_eq!(result, Ok(()));
 }
 
-#[test]
-fn invalid_deposit_invalid_pub_key() {
+#[tokio::test]
+async fn invalid_deposit_invalid_pub_key() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut state = harness.get_current_state();
 
     let (deposits, state) =
         harness.make_deposits(&mut state, 1, Some(PublicKeyBytes::empty()), None);
     let deposits = VariableList::from(deposits);
 
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     *head_block.to_mut().body_mut().deposits_mut() = deposits;
 
     let result = process_operations::process_deposits(state, head_block.body().deposits(), &spec);
@@ -324,13 +370,19 @@ fn invalid_deposit_invalid_pub_key() {
     assert_eq!(result, Ok(()));
 }
 
-#[test]
-fn invalid_attestation_no_committee_for_index() {
+#[tokio::test]
+async fn invalid_attestation_no_committee_for_index() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     head_block.to_mut().body_mut().attestations_mut()[0]
         .data
         .index += 1;
@@ -352,13 +404,19 @@ fn invalid_attestation_no_committee_for_index() {
     );
 }
 
-#[test]
-fn invalid_attestation_wrong_justified_checkpoint() {
+#[tokio::test]
+async fn invalid_attestation_wrong_justified_checkpoint() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     let old_justified_checkpoint = head_block.body().attestations()[0].data.source;
     let mut new_justified_checkpoint = old_justified_checkpoint;
     new_justified_checkpoint.epoch += Epoch::new(1);
@@ -389,13 +447,19 @@ fn invalid_attestation_wrong_justified_checkpoint() {
     );
 }
 
-#[test]
-fn invalid_attestation_bad_aggregation_bitfield_len() {
+#[tokio::test]
+async fn invalid_attestation_bad_aggregation_bitfield_len() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     head_block.to_mut().body_mut().attestations_mut()[0].aggregation_bits =
         Bitfield::with_capacity(spec.target_committee_size).unwrap();
 
@@ -416,13 +480,19 @@ fn invalid_attestation_bad_aggregation_bitfield_len() {
     );
 }
 
-#[test]
-fn invalid_attestation_bad_signature() {
+#[tokio::test]
+async fn invalid_attestation_bad_signature() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, 97); // minimal number of required validators for this test
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, 97).await; // minimal number of required validators for this test
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     head_block.to_mut().body_mut().attestations_mut()[0].signature = AggregateSignature::empty();
 
     let result = process_operations::process_attestations(
@@ -444,13 +514,19 @@ fn invalid_attestation_bad_signature() {
     );
 }
 
-#[test]
-fn invalid_attestation_included_too_early() {
+#[tokio::test]
+async fn invalid_attestation_included_too_early() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     let new_attesation_slot = head_block.body().attestations()[0].data.slot
         + Slot::new(MainnetEthSpec::slots_per_epoch());
     head_block.to_mut().body_mut().attestations_mut()[0]
@@ -479,14 +555,20 @@ fn invalid_attestation_included_too_early() {
     );
 }
 
-#[test]
-fn invalid_attestation_included_too_late() {
+#[tokio::test]
+async fn invalid_attestation_included_too_late() {
     let spec = MainnetEthSpec::default_spec();
     // note to maintainer: might need to increase validator count if we get NoCommittee
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     let new_attesation_slot = head_block.body().attestations()[0].data.slot
         - Slot::new(MainnetEthSpec::slots_per_epoch());
     head_block.to_mut().body_mut().attestations_mut()[0]
@@ -512,14 +594,20 @@ fn invalid_attestation_included_too_late() {
     );
 }
 
-#[test]
-fn invalid_attestation_target_epoch_slot_mismatch() {
+#[tokio::test]
+async fn invalid_attestation_target_epoch_slot_mismatch() {
     let spec = MainnetEthSpec::default_spec();
     // note to maintainer: might need to increase validator count if we get NoCommittee
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut state = harness.get_current_state();
-    let mut head_block = harness.chain.head_beacon_block().unwrap().deconstruct().0;
+    let mut head_block = harness
+        .chain
+        .head_beacon_block()
+        .as_ref()
+        .clone()
+        .deconstruct()
+        .0;
     head_block.to_mut().body_mut().attestations_mut()[0]
         .data
         .target
@@ -544,10 +632,10 @@ fn invalid_attestation_target_epoch_slot_mismatch() {
     );
 }
 
-#[test]
-fn valid_insert_attester_slashing() {
+#[tokio::test]
+async fn valid_insert_attester_slashing() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let attester_slashing = harness.make_attester_slashing(vec![1, 2]);
 
@@ -563,10 +651,10 @@ fn valid_insert_attester_slashing() {
     assert_eq!(result, Ok(()));
 }
 
-#[test]
-fn invalid_attester_slashing_not_slashable() {
+#[tokio::test]
+async fn invalid_attester_slashing_not_slashable() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut attester_slashing = harness.make_attester_slashing(vec![1, 2]);
     attester_slashing.attestation_1 = attester_slashing.attestation_2.clone();
@@ -589,10 +677,10 @@ fn invalid_attester_slashing_not_slashable() {
     );
 }
 
-#[test]
-fn invalid_attester_slashing_1_invalid() {
+#[tokio::test]
+async fn invalid_attester_slashing_1_invalid() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut attester_slashing = harness.make_attester_slashing(vec![1, 2]);
     attester_slashing.attestation_1.attesting_indices = VariableList::from(vec![2, 1]);
@@ -618,10 +706,10 @@ fn invalid_attester_slashing_1_invalid() {
     );
 }
 
-#[test]
-fn invalid_attester_slashing_2_invalid() {
+#[tokio::test]
+async fn invalid_attester_slashing_2_invalid() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut attester_slashing = harness.make_attester_slashing(vec![1, 2]);
     attester_slashing.attestation_2.attesting_indices = VariableList::from(vec![2, 1]);
@@ -647,10 +735,10 @@ fn invalid_attester_slashing_2_invalid() {
     );
 }
 
-#[test]
-fn valid_insert_proposer_slashing() {
+#[tokio::test]
+async fn valid_insert_proposer_slashing() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let proposer_slashing = harness.make_proposer_slashing(1);
     let mut state = harness.get_current_state();
     let result = process_operations::process_proposer_slashings(
@@ -663,10 +751,10 @@ fn valid_insert_proposer_slashing() {
     assert!(result.is_ok());
 }
 
-#[test]
-fn invalid_proposer_slashing_proposals_identical() {
+#[tokio::test]
+async fn invalid_proposer_slashing_proposals_identical() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut proposer_slashing = harness.make_proposer_slashing(1);
     proposer_slashing.signed_header_1.message = proposer_slashing.signed_header_2.message.clone();
@@ -689,10 +777,10 @@ fn invalid_proposer_slashing_proposals_identical() {
     );
 }
 
-#[test]
-fn invalid_proposer_slashing_proposer_unknown() {
+#[tokio::test]
+async fn invalid_proposer_slashing_proposer_unknown() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let mut proposer_slashing = harness.make_proposer_slashing(1);
     proposer_slashing.signed_header_1.message.proposer_index = 3_141_592;
@@ -716,10 +804,10 @@ fn invalid_proposer_slashing_proposer_unknown() {
     );
 }
 
-#[test]
-fn invalid_proposer_slashing_duplicate_slashing() {
+#[tokio::test]
+async fn invalid_proposer_slashing_duplicate_slashing() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
 
     let proposer_slashing = harness.make_proposer_slashing(1);
     let mut state = harness.get_current_state();
@@ -747,10 +835,10 @@ fn invalid_proposer_slashing_duplicate_slashing() {
     );
 }
 
-#[test]
-fn invalid_bad_proposal_1_signature() {
+#[tokio::test]
+async fn invalid_bad_proposal_1_signature() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut proposer_slashing = harness.make_proposer_slashing(1);
     proposer_slashing.signed_header_1.signature = Signature::empty();
     let mut state = harness.get_current_state();
@@ -771,10 +859,10 @@ fn invalid_bad_proposal_1_signature() {
     );
 }
 
-#[test]
-fn invalid_bad_proposal_2_signature() {
+#[tokio::test]
+async fn invalid_bad_proposal_2_signature() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut proposer_slashing = harness.make_proposer_slashing(1);
     proposer_slashing.signed_header_2.signature = Signature::empty();
     let mut state = harness.get_current_state();
@@ -795,10 +883,10 @@ fn invalid_bad_proposal_2_signature() {
     );
 }
 
-#[test]
-fn invalid_proposer_slashing_proposal_epoch_mismatch() {
+#[tokio::test]
+async fn invalid_proposer_slashing_proposal_epoch_mismatch() {
     let spec = MainnetEthSpec::default_spec();
-    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT);
+    let harness = get_harness::<MainnetEthSpec>(EPOCH_OFFSET, VALIDATOR_COUNT).await;
     let mut proposer_slashing = harness.make_proposer_slashing(1);
     proposer_slashing.signed_header_1.message.slot = Slot::new(0);
     proposer_slashing.signed_header_2.message.slot = Slot::new(128);