Optimize validator duties (#2243)

## Issue Addressed

Closes #2052

## Proposed Changes

- Refactor the attester/proposer duties endpoints in the BN
    - Performance improvements
    - Fixes some potential inconsistencies with the dependent root fields.
    - Removes `http_api::beacon_proposer_cache` and just uses the one on the `BeaconChain` instead.
    - Move the code for the proposer/attester duties endpoints into separate files, for readability.
- Refactor the `DutiesService` in the VC
    - Required to reduce the delay on broadcasting new blocks.
    - Gets rid of the `ValidatorDuty` shim struct that came about when we adopted the standard API.
    - Separate block/attestation duty tasks so that they don't block each other when one is slow.
- In the VC, use `PublicKeyBytes` to represent validators instead of `PublicKey`. `PublicKey` is a legit crypto object whilst `PublicKeyBytes` is just a byte-array, it's much faster to clone/hash `PublicKeyBytes` and this change has had a significant impact on runtimes.
    - Unfortunately this has created lots of dust changes.
 - In the BN, store `PublicKeyBytes` in the `beacon_proposer_cache` and allow access to them. The HTTP API always sends `PublicKeyBytes` over the wire and the conversion from `PublicKey` -> `PublickeyBytes` is non-trivial, especially when queries have 100s/1000s of validators (like Pyrmont).
 - Add the `state_processing::state_advance` mod which dedups a lot of the "apply `n` skip slots to the state" code.
    - This also fixes a bug with some functions which were failing to include a state root as per [this comment](072695284f/consensus/state_processing/src/state_advance.rs (L69-L74)). I couldn't find any instance of this bug that resulted in anything more severe than keying a shuffling cache by the wrong block root.
 - Swap the VC block service to use `mpsc` from `tokio` instead of `futures`. This is consistent with the rest of the code base.
    
~~This PR *reduces* the size of the codebase 🎉~~ It *used* to reduce the size of the code base before I added more comments. 

## Observations on Prymont

- Proposer duties times down from peaks of 450ms to consistent <1ms.
- Current epoch attester duties times down from >1s peaks to a consistent 20-30ms.
- Block production down from +600ms to 100-200ms.

## Additional Info

- ~~Blocked on #2241~~
- ~~Blocked on #2234~~

## TODO

- [x] ~~Refactor this into some smaller PRs?~~ Leaving this as-is for now.
- [x] Address `per_slot_processing` roots.
- [x] Investigate slow next epoch times. Not getting added to cache on block processing?
- [x] Consider [this](072695284f/beacon_node/store/src/hot_cold_store.rs (L811-L812)) in the scenario of replacing the state roots


Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>

consensus/fork_choice/tests/tests.rs

@@ -183,8 +183,13 @@ impl ForkChoiceTest {
                 break;
             }
             if let Ok(block_hash) = self.harness.process_block_result(block.clone()) {
-                self.harness
-                    .attest_block(&state, block_hash, &block, &validators);
+                self.harness.attest_block(
+                    &state,
+                    block.state_root(),
+                    block_hash,
+                    &block,
+                    &validators,
+                );
                 self.harness.advance_slot();
             } else {
                 return Err(self);

consensus/state_processing/src/lib.rs

@@ -9,6 +9,7 @@ pub mod genesis;
 pub mod per_block_processing;
 pub mod per_epoch_processing;
 pub mod per_slot_processing;
+pub mod state_advance;
 pub mod test_utils;
 pub mod verify_operation;
 

consensus/state_processing/src/state_advance.rs

@@ -0,0 +1,105 @@
+//! This module contains functions for advancing a `BeaconState` forward some number of slots
+//! without blocks (i.e., skip slots).
+//!
+//! These functions are not in the specification, however they're defined here to reduce code
+//! duplication and protect against some easy-to-make mistakes when performing state advances.
+
+use crate::*;
+use types::{BeaconState, ChainSpec, EthSpec, Hash256, Slot};
+
+#[derive(Debug, PartialEq)]
+pub enum Error {
+    BadTargetSlot { target_slot: Slot, state_slot: Slot },
+    PerSlotProcessing(per_slot_processing::Error),
+    StateRootNotProvided,
+}
+
+/// Advances the `state` to the given `target_slot`, assuming that there were no blocks between
+/// these slots.
+///
+/// ## Errors
+///
+/// - If `state.slot > target_slot`, an error will be returned.
+///
+/// ## Notes
+///
+/// This state advance method is "complete"; it outputs a perfectly valid `BeaconState` and doesn't
+/// do anything hacky like the "partial" method (see `partial_state_advance`).
+pub fn complete_state_advance<T: EthSpec>(
+    state: &mut BeaconState<T>,
+    mut state_root_opt: Option<Hash256>,
+    target_slot: Slot,
+    spec: &ChainSpec,
+) -> Result<(), Error> {
+    check_target_slot(state.slot, target_slot)?;
+
+    while state.slot < target_slot {
+        // Use the initial state root on the first iteration of the loop, then use `None`  for any
+        // future iterations.
+        let state_root_opt = state_root_opt.take();
+
+        per_slot_processing(state, state_root_opt, spec).map_err(Error::PerSlotProcessing)?;
+    }
+
+    Ok(())
+}
+
+/// Advances the `state` to the given `target_slot`, assuming that there were no blocks between
+/// these slots.
+///
+/// This is a "partial" state advance which outputs an **invalid** `BeaconState`. The state is
+/// invalid because the intermediate state roots are not computed. Avoiding computing state roots
+/// saves *a lot* of compute time and can be a useful optimization when a state only needs to be
+/// advanced to obtain proposer/attester shuffling as they are indifferent to state roots.
+///
+/// For clarity, **be careful with this function as it produces invalid states**.
+///
+/// ## Errors
+///
+/// - If `state.slot > target_slot`, an error will be returned.
+/// - If `state_root_opt.is_none()` but the latest block header requires a state root.
+pub fn partial_state_advance<T: EthSpec>(
+    state: &mut BeaconState<T>,
+    state_root_opt: Option<Hash256>,
+    target_slot: Slot,
+    spec: &ChainSpec,
+) -> Result<(), Error> {
+    check_target_slot(state.slot, target_slot)?;
+
+    // The only time that a state root is mandatory is if a block has been applied to the state
+    // without it yet being advanced another slot.
+    //
+    // Failing to provide a state root in this scenario would result in corrupting the
+    // `state.block_roots` array, since the `state.latest_block_header` would contain an invalid
+    // (all-zeros) state root.
+    let mut initial_state_root = Some(if state.slot > state.latest_block_header.slot {
+        state_root_opt.unwrap_or_else(Hash256::zero)
+    } else {
+        state_root_opt.ok_or(Error::StateRootNotProvided)?
+    });
+
+    while state.slot < target_slot {
+        // Use the initial state root on the first iteration of the loop, then use `[0; 32]` for any
+        // later iterations.
+        //
+        // Failing to provide the correct state root on the initial iteration may result in
+        // corrupting the `state.block_roots` array since the latest block header may not be updated
+        // with the correct state root.
+        let state_root = initial_state_root.take().unwrap_or_else(Hash256::zero);
+
+        per_slot_processing(state, Some(state_root), spec).map_err(Error::PerSlotProcessing)?;
+    }
+
+    Ok(())
+}
+
+fn check_target_slot(state_slot: Slot, target_slot: Slot) -> Result<(), Error> {
+    if state_slot > target_slot {
+        Err(Error::BadTargetSlot {
+            target_slot,
+            state_slot,
+        })
+    } else {
+        Ok(())
+    }
+}

consensus/types/src/beacon_state.rs

@@ -444,6 +444,62 @@ impl<T: EthSpec> BeaconState<T> {
         cache.get_all_beacon_committees()
     }
 
+    /// Returns the block root which decided the proposer shuffling for the current epoch. This root
+    /// can be used to key this proposer shuffling.
+    ///
+    /// ## Notes
+    ///
+    /// The `block_root` covers the one-off scenario where the genesis block decides its own
+    /// shuffling. It should be set to the latest block applied to `self` or the genesis block root.
+    pub fn proposer_shuffling_decision_root(&self, block_root: Hash256) -> Result<Hash256, Error> {
+        let decision_slot = self.proposer_shuffling_decision_slot();
+        if self.slot == decision_slot {
+            Ok(block_root)
+        } else {
+            self.get_block_root(decision_slot).map(|root| *root)
+        }
+    }
+
+    /// Returns the slot at which the proposer shuffling was decided. The block root at this slot
+    /// can be used to key the proposer shuffling for the current epoch.
+    fn proposer_shuffling_decision_slot(&self) -> Slot {
+        self.current_epoch()
+            .start_slot(T::slots_per_epoch())
+            .saturating_sub(1_u64)
+    }
+
+    /// Returns the block root which decided the attester shuffling for the given `relative_epoch`.
+    /// This root can be used to key that attester shuffling.
+    ///
+    /// ## Notes
+    ///
+    /// The `block_root` covers the one-off scenario where the genesis block decides its own
+    /// shuffling. It should be set to the latest block applied to `self` or the genesis block root.
+    pub fn attester_shuffling_decision_root(
+        &self,
+        block_root: Hash256,
+        relative_epoch: RelativeEpoch,
+    ) -> Result<Hash256, Error> {
+        let decision_slot = self.attester_shuffling_decision_slot(relative_epoch);
+        if self.slot == decision_slot {
+            Ok(block_root)
+        } else {
+            self.get_block_root(decision_slot).map(|root| *root)
+        }
+    }
+
+    /// Returns the slot at which the proposer shuffling was decided. The block root at this slot
+    /// can be used to key the proposer shuffling for the current epoch.
+    fn attester_shuffling_decision_slot(&self, relative_epoch: RelativeEpoch) -> Slot {
+        match relative_epoch {
+            RelativeEpoch::Next => self.current_epoch(),
+            RelativeEpoch::Current => self.previous_epoch(),
+            RelativeEpoch::Previous => self.previous_epoch().saturating_sub(1_u64),
+        }
+        .start_slot(T::slots_per_epoch())
+        .saturating_sub(1_u64)
+    }
+
     /// Compute the proposer (not necessarily for the Beacon chain) from a list of indices.
     ///
     /// Spec v0.12.1

consensus/types/src/shuffling_id.rs

@@ -35,16 +35,8 @@ impl AttestationShufflingId {
     ) -> Result<Self, BeaconStateError> {
         let shuffling_epoch = relative_epoch.into_epoch(state.current_epoch());
 
-        let shuffling_decision_slot = shuffling_epoch
-            .saturating_sub(1_u64)
-            .start_slot(E::slots_per_epoch())
-            .saturating_sub(1_u64);
-
-        let shuffling_decision_block = if state.slot == shuffling_decision_slot {
-            block_root
-        } else {
-            *state.get_block_root(shuffling_decision_slot)?
-        };
+        let shuffling_decision_block =
+            state.attester_shuffling_decision_root(block_root, relative_epoch)?;
 
         Ok(Self {
             shuffling_epoch,