Use async code when interacting with EL (#3244)

## Overview

This rather extensive PR achieves two primary goals:

1. Uses the finalized/justified checkpoints of fork choice (FC), rather than that of the head state.
2. Refactors fork choice, block production and block processing to `async` functions.

Additionally, it achieves:

- Concurrent forkchoice updates to the EL and cache pruning after a new head is selected.
- Concurrent "block packing" (attestations, etc) and execution payload retrieval during block production.
- Concurrent per-block-processing and execution payload verification during block processing.
- The `Arc`-ification of `SignedBeaconBlock` during block processing (it's never mutated, so why not?):
    - I had to do this to deal with sending blocks into spawned tasks.
    - Previously we were cloning the beacon block at least 2 times during each block processing, these clones are either removed or turned into cheaper `Arc` clones.
    - We were also `Box`-ing and un-`Box`-ing beacon blocks as they moved throughout the networking crate. This is not a big deal, but it's nice to avoid shifting things between the stack and heap.
    - Avoids cloning *all the blocks* in *every chain segment* during sync.
    - It also has the potential to clean up our code where we need to pass an *owned* block around so we can send it back in the case of an error (I didn't do much of this, my PR is already big enough 😅)
- The `BeaconChain::HeadSafetyStatus` struct was removed. It was an old relic from prior merge specs.

For motivation for this change, see https://github.com/sigp/lighthouse/pull/3244#issuecomment-1160963273

## Changes to `canonical_head` and `fork_choice`

Previously, the `BeaconChain` had two separate fields:

```
canonical_head: RwLock<Snapshot>,
fork_choice: RwLock<BeaconForkChoice>
```

Now, we have grouped these values under a single struct:

```
canonical_head: CanonicalHead {
  cached_head: RwLock<Arc<Snapshot>>,
  fork_choice: RwLock<BeaconForkChoice>
} 
```

Apart from ergonomics, the only *actual* change here is wrapping the canonical head snapshot in an `Arc`. This means that we no longer need to hold the `cached_head` (`canonical_head`, in old terms) lock when we want to pull some values from it. This was done to avoid deadlock risks by preventing functions from acquiring (and holding) the `cached_head` and `fork_choice` locks simultaneously.

## Breaking Changes

### The `state` (root) field in the `finalized_checkpoint` SSE event

Consider the scenario where epoch `n` is just finalized, but `start_slot(n)` is skipped. There are two state roots we might in the `finalized_checkpoint` SSE event:

1. The state root of the finalized block, which is `get_block(finalized_checkpoint.root).state_root`.
4. The state root at slot of `start_slot(n)`, which would be the state from (1), but "skipped forward" through any skip slots.

Previously, Lighthouse would choose (2). However, we can see that when [Teku generates that event](de2b2801c8/data/beaconrestapi/src/main/java/tech/pegasys/teku/beaconrestapi/handlers/v1/events/EventSubscriptionManager.java (L171-L182)) it uses [`getStateRootFromBlockRoot`](de2b2801c8/data/provider/src/main/java/tech/pegasys/teku/api/ChainDataProvider.java (L336-L341)) which uses (1).

I have switched Lighthouse from (2) to (1). I think it's a somewhat arbitrary choice between the two, where (1) is easier to compute and is consistent with Teku.

## Notes for Reviewers

I've renamed `BeaconChain::fork_choice` to `BeaconChain::recompute_head`. Doing this helped ensure I broke all previous uses of fork choice and I also find it more descriptive. It describes an action and can't be confused with trying to get a reference to the `ForkChoice` struct.

I've changed the ordering of SSE events when a block is received. It used to be `[block, finalized, head]` and now it's `[block, head, finalized]`. It was easier this way and I don't think we were making any promises about SSE event ordering so it's not "breaking".

I've made it so fork choice will run when it's first constructed. I did this because I wanted to have a cached version of the last call to `get_head`. Ensuring `get_head` has been run *at least once* means that the cached values doesn't need to wrapped in an `Option`. This was fairly simple, it just involved passing a `slot` to the constructor so it knows *when* it's being run. When loading a fork choice from the store and a slot clock isn't handy I've just used the `slot` that was saved in the `fork_choice_store`. That seems like it would be a faithful representation of the slot when we saved it.

I added the `genesis_time: u64` to the `BeaconChain`. It's small, constant and nice to have around.

Since we're using FC for the fin/just checkpoints, we no longer get the `0x00..00` roots at genesis. You can see I had to remove a work-around in `ef-tests` here: b56be3bc2. I can't find any reason why this would be an issue, if anything I think it'll be better since the genesis-alias has caught us out a few times (0x00..00 isn't actually a real root). Edit: I did find a case where the `network` expected the 0x00..00 alias and patched it here: 3f26ac3e2.

You'll notice a lot of changes in tests. Generally, tests should be functionally equivalent. Here are the things creating the most diff-noise in tests:
- Changing tests to be `tokio::async` tests.
- Adding `.await` to fork choice, block processing and block production functions.
- Refactor of the `canonical_head` "API" provided by the `BeaconChain`. E.g., `chain.canonical_head.cached_head()` instead of `chain.canonical_head.read()`.
- Wrapping `SignedBeaconBlock` in an `Arc`.
- In the `beacon_chain/tests/block_verification`, we can't use the `lazy_static` `CHAIN_SEGMENT` variable anymore since it's generated with an async function. We just generate it in each test, not so efficient but hopefully insignificant.

I had to disable `rayon` concurrent tests in the `fork_choice` tests. This is because the use of `rayon` and `block_on` was causing a panic.

Co-authored-by: Mac L <mjladson@pm.me>

consensus/fork_choice/src/fork_choice.rs

@@ -5,7 +5,7 @@ use std::cmp::Ordering;
 use std::marker::PhantomData;
 use std::time::Duration;
 use types::{
-    consts::merge::INTERVALS_PER_SLOT, AttestationShufflingId, BeaconBlock, BeaconState,
+    consts::merge::INTERVALS_PER_SLOT, AttestationShufflingId, BeaconBlockRef, BeaconState,
     BeaconStateError, ChainSpec, Checkpoint, Epoch, EthSpec, ExecPayload, ExecutionBlockHash,
     Hash256, IndexedAttestation, RelativeEpoch, SignedBeaconBlock, Slot,
 };
@@ -248,6 +248,7 @@ fn dequeue_attestations(
 /// Equivalent to the `is_from_block` `bool` in:
 ///
 /// https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/fork-choice.md#validate_on_attestation
+#[derive(Clone, Copy)]
 pub enum AttestationFromBlock {
     True,
     False,
@@ -261,6 +262,13 @@ pub struct ForkchoiceUpdateParameters {
     pub finalized_hash: Option<ExecutionBlockHash>,
 }
 
+#[derive(Clone, Copy, Debug, PartialEq)]
+pub struct ForkChoiceView {
+    pub head_block_root: Hash256,
+    pub justified_checkpoint: Checkpoint,
+    pub finalized_checkpoint: Checkpoint,
+}
+
 /// Provides an implementation of "Ethereum 2.0 Phase 0 -- Beacon Chain Fork Choice":
 ///
 /// https://github.com/ethereum/eth2.0-specs/blob/v0.12.1/specs/phase0/fork-choice.md#ethereum-20-phase-0----beacon-chain-fork-choice
@@ -279,7 +287,9 @@ pub struct ForkChoice<T, E> {
     /// Attestations that arrived at the current slot and must be queued for later processing.
     queued_attestations: Vec<QueuedAttestation>,
     /// Stores a cache of the values required to be sent to the execution layer.
-    forkchoice_update_parameters: Option<ForkchoiceUpdateParameters>,
+    forkchoice_update_parameters: ForkchoiceUpdateParameters,
+    /// The most recent result of running `Self::get_head`.
+    head_block_root: Hash256,
     _phantom: PhantomData<E>,
 }
 
@@ -306,6 +316,8 @@ where
         anchor_block_root: Hash256,
         anchor_block: &SignedBeaconBlock<E>,
         anchor_state: &BeaconState<E>,
+        current_slot: Option<Slot>,
+        spec: &ChainSpec,
     ) -> Result<Self, Error<T::Error>> {
         // Sanity check: the anchor must lie on an epoch boundary.
         if anchor_block.slot() % E::slots_per_epoch() != 0 {
@@ -340,6 +352,9 @@ where
             },
         );
 
+        // If the current slot is not provided, use the value that was last provided to the store.
+        let current_slot = current_slot.unwrap_or_else(|| fc_store.get_current_slot());
+
         let proto_array = ProtoArrayForkChoice::new(
             finalized_block_slot,
             finalized_block_state_root,
@@ -350,15 +365,28 @@ where
             execution_status,
         )?;
 
-        Ok(Self {
+        let mut fork_choice = Self {
             fc_store,
             proto_array,
             queued_attestations: vec![],
-            forkchoice_update_parameters: None,
+            // This will be updated during the next call to `Self::get_head`.
+            forkchoice_update_parameters: ForkchoiceUpdateParameters {
+                head_hash: None,
+                finalized_hash: None,
+                head_root: Hash256::zero(),
+            },
+            // This will be updated during the next call to `Self::get_head`.
+            head_block_root: Hash256::zero(),
             _phantom: PhantomData,
-        })
+        };
+
+        // Ensure that `fork_choice.head_block_root` is updated.
+        fork_choice.get_head(current_slot, spec)?;
+
+        Ok(fork_choice)
     }
 
+    /*
     /// Instantiates `Self` from some existing components.
     ///
     /// This is useful if the existing components have been loaded from disk after a process
@@ -376,13 +404,13 @@ where
             _phantom: PhantomData,
         }
     }
+    */
 
     /// Returns cached information that can be used to issue a `forkchoiceUpdated` message to an
     /// execution engine.
     ///
-    /// These values are updated each time `Self::get_head` is called. May return `None` if
-    /// `Self::get_head` has not yet been called.
-    pub fn get_forkchoice_update_parameters(&self) -> Option<ForkchoiceUpdateParameters> {
+    /// These values are updated each time `Self::get_head` is called.
+    pub fn get_forkchoice_update_parameters(&self) -> ForkchoiceUpdateParameters {
         self.forkchoice_update_parameters
     }
 
@@ -455,6 +483,8 @@ where
             spec,
         )?;
 
+        self.head_block_root = head_root;
+
         // Cache some values for the next forkchoiceUpdate call to the execution layer.
         let head_hash = self
             .get_block(&head_root)
@@ -463,15 +493,35 @@ where
         let finalized_hash = self
             .get_block(&finalized_root)
             .and_then(|b| b.execution_status.block_hash());
-        self.forkchoice_update_parameters = Some(ForkchoiceUpdateParameters {
+        self.forkchoice_update_parameters = ForkchoiceUpdateParameters {
             head_root,
             head_hash,
             finalized_hash,
-        });
+        };
 
         Ok(head_root)
     }
 
+    /// Return information about:
+    ///
+    /// - The LMD head of the chain.
+    /// - The FFG checkpoints.
+    ///
+    /// The information is "cached" since the last call to `Self::get_head`.
+    ///
+    /// ## Notes
+    ///
+    /// The finalized/justified checkpoints are determined from the fork choice store. Therefore,
+    /// it's possible that the state corresponding to `get_state(get_block(head_block_root))` will
+    /// have *differing* finalized and justified information.
+    pub fn cached_fork_choice_view(&self) -> ForkChoiceView {
+        ForkChoiceView {
+            head_block_root: self.head_block_root,
+            justified_checkpoint: self.justified_checkpoint(),
+            finalized_checkpoint: self.finalized_checkpoint(),
+        }
+    }
+
     /// Returns `true` if the given `store` should be updated to set
     /// `state.current_justified_checkpoint` its `justified_checkpoint`.
     ///
@@ -566,7 +616,7 @@ where
     pub fn on_block<Payload: ExecPayload<E>>(
         &mut self,
         current_slot: Slot,
-        block: &BeaconBlock<E, Payload>,
+        block: BeaconBlockRef<E, Payload>,
         block_root: Hash256,
         block_delay: Duration,
         state: &BeaconState<E>,
@@ -966,6 +1016,11 @@ where
         }
     }
 
+    /// Returns the weight for the given block root.
+    pub fn get_block_weight(&self, block_root: &Hash256) -> Option<u64> {
+        self.proto_array.get_weight(block_root)
+    }
+
     /// Returns the `ProtoBlock` for the justified checkpoint.
     ///
     /// ## Notes
@@ -995,6 +1050,39 @@ where
             .is_descendant(self.fc_store.finalized_checkpoint().root, block_root)
     }
 
+    /// Returns `Ok(true)` if `block_root` has been imported optimistically. That is, the
+    /// execution payload has not been verified.
+    ///
+    /// Returns `Ok(false)` if `block_root`'s execution payload has been verfied, if it is a
+    /// pre-Bellatrix block or if it is before the PoW terminal block.
+    ///
+    /// In the case where the block could not be found in fork-choice, it returns the
+    /// `execution_status` of the current finalized block.
+    ///
+    /// This function assumes the `block_root` exists.
+    pub fn is_optimistic_block(&self, block_root: &Hash256) -> Result<bool, Error<T::Error>> {
+        if let Some(status) = self.get_block_execution_status(block_root) {
+            Ok(status.is_optimistic())
+        } else {
+            Ok(self.get_finalized_block()?.execution_status.is_optimistic())
+        }
+    }
+
+    /// The same as `is_optimistic_block` but does not fallback to `self.get_finalized_block`
+    /// when the block cannot be found.
+    ///
+    /// Intended to be used when checking if the head has been imported optimistically.
+    pub fn is_optimistic_block_no_fallback(
+        &self,
+        block_root: &Hash256,
+    ) -> Result<bool, Error<T::Error>> {
+        if let Some(status) = self.get_block_execution_status(block_root) {
+            Ok(status.is_optimistic())
+        } else {
+            Err(Error::MissingProtoArrayBlock(*block_root))
+        }
+    }
+
     /// Returns `Ok(false)` if a block is not viable to be imported optimistically.
     ///
     /// ## Notes
@@ -1109,17 +1197,31 @@ where
     pub fn from_persisted(
         persisted: PersistedForkChoice,
         fc_store: T,
+        spec: &ChainSpec,
     ) -> Result<Self, Error<T::Error>> {
         let proto_array = ProtoArrayForkChoice::from_bytes(&persisted.proto_array_bytes)
             .map_err(Error::InvalidProtoArrayBytes)?;
 
-        Ok(Self {
+        let current_slot = fc_store.get_current_slot();
+
+        let mut fork_choice = Self {
             fc_store,
             proto_array,
             queued_attestations: persisted.queued_attestations,
-            forkchoice_update_parameters: None,
+            // Will be updated in the following call to `Self::get_head`.
+            forkchoice_update_parameters: ForkchoiceUpdateParameters {
+                head_hash: None,
+                finalized_hash: None,
+                head_root: Hash256::zero(),
+            },
+            // Will be updated in the following call to `Self::get_head`.
+            head_block_root: Hash256::zero(),
             _phantom: PhantomData,
-        })
+        };
+
+        fork_choice.get_head(current_slot, spec)?;
+
+        Ok(fork_choice)
     }
 
     /// Takes a snapshot of `Self` and stores it in `PersistedForkChoice`, allowing this struct to

consensus/fork_choice/src/fork_choice_store.rs

@@ -1,4 +1,4 @@
-use types::{BeaconBlock, BeaconState, Checkpoint, EthSpec, ExecPayload, Hash256, Slot};
+use types::{BeaconBlockRef, BeaconState, Checkpoint, EthSpec, ExecPayload, Hash256, Slot};
 
 /// Approximates the `Store` in "Ethereum 2.0 Phase 0 -- Beacon Chain Fork Choice":
 ///
@@ -33,7 +33,7 @@ pub trait ForkChoiceStore<T: EthSpec>: Sized {
     /// choice. Allows the implementer to performing caching or other housekeeping duties.
     fn on_verified_block<Payload: ExecPayload<T>>(
         &mut self,
-        block: &BeaconBlock<T, Payload>,
+        block: BeaconBlockRef<T, Payload>,
         block_root: Hash256,
         state: &BeaconState<T>,
     ) -> Result<(), Self::Error>;

consensus/fork_choice/src/lib.rs

@@ -2,8 +2,9 @@ mod fork_choice;
 mod fork_choice_store;
 
 pub use crate::fork_choice::{
-    AttestationFromBlock, Error, ForkChoice, InvalidAttestation, InvalidBlock,
-    PayloadVerificationStatus, PersistedForkChoice, QueuedAttestation,
+    AttestationFromBlock, Error, ForkChoice, ForkChoiceView, ForkchoiceUpdateParameters,
+    InvalidAttestation, InvalidBlock, PayloadVerificationStatus, PersistedForkChoice,
+    QueuedAttestation,
 };
 pub use fork_choice_store::ForkChoiceStore;
 pub use proto_array::{Block as ProtoBlock, ExecutionStatus, InvalidationOperation};