Optimize validator duties (#2243)

## Issue Addressed

Closes #2052

## Proposed Changes

- Refactor the attester/proposer duties endpoints in the BN
    - Performance improvements
    - Fixes some potential inconsistencies with the dependent root fields.
    - Removes `http_api::beacon_proposer_cache` and just uses the one on the `BeaconChain` instead.
    - Move the code for the proposer/attester duties endpoints into separate files, for readability.
- Refactor the `DutiesService` in the VC
    - Required to reduce the delay on broadcasting new blocks.
    - Gets rid of the `ValidatorDuty` shim struct that came about when we adopted the standard API.
    - Separate block/attestation duty tasks so that they don't block each other when one is slow.
- In the VC, use `PublicKeyBytes` to represent validators instead of `PublicKey`. `PublicKey` is a legit crypto object whilst `PublicKeyBytes` is just a byte-array, it's much faster to clone/hash `PublicKeyBytes` and this change has had a significant impact on runtimes.
    - Unfortunately this has created lots of dust changes.
 - In the BN, store `PublicKeyBytes` in the `beacon_proposer_cache` and allow access to them. The HTTP API always sends `PublicKeyBytes` over the wire and the conversion from `PublicKey` -> `PublickeyBytes` is non-trivial, especially when queries have 100s/1000s of validators (like Pyrmont).
 - Add the `state_processing::state_advance` mod which dedups a lot of the "apply `n` skip slots to the state" code.
    - This also fixes a bug with some functions which were failing to include a state root as per [this comment](072695284f/consensus/state_processing/src/state_advance.rs (L69-L74)). I couldn't find any instance of this bug that resulted in anything more severe than keying a shuffling cache by the wrong block root.
 - Swap the VC block service to use `mpsc` from `tokio` instead of `futures`. This is consistent with the rest of the code base.
    
~~This PR *reduces* the size of the codebase 🎉~~ It *used* to reduce the size of the code base before I added more comments. 

## Observations on Prymont

- Proposer duties times down from peaks of 450ms to consistent <1ms.
- Current epoch attester duties times down from >1s peaks to a consistent 20-30ms.
- Block production down from +600ms to 100-200ms.

## Additional Info

- ~~Blocked on #2241~~
- ~~Blocked on #2234~~

## TODO

- [x] ~~Refactor this into some smaller PRs?~~ Leaving this as-is for now.
- [x] Address `per_slot_processing` roots.
- [x] Investigate slow next epoch times. Not getting added to cache on block processing?
- [x] Consider [this](072695284f/beacon_node/store/src/hot_cold_store.rs (L811-L812)) in the scenario of replacing the state roots


Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>

beacon_node/beacon_chain/src/validator_pubkey_cache.rs

@@ -23,6 +23,7 @@ use types::{BeaconState, Hash256, PublicKey, PublicKeyBytes};
 pub struct ValidatorPubkeyCache<T: BeaconChainTypes> {
     pubkeys: Vec<PublicKey>,
     indices: HashMap<PublicKeyBytes, usize>,
+    pubkey_bytes: Vec<PublicKeyBytes>,
     backing: PubkeyCacheBacking<T>,
 }
 
@@ -46,6 +47,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         let mut cache = Self {
             pubkeys: vec![],
             indices: HashMap::new(),
+            pubkey_bytes: vec![],
             backing: PubkeyCacheBacking::Database(store),
         };
 
@@ -58,12 +60,14 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
     pub fn load_from_store(store: BeaconStore<T>) -> Result<Self, BeaconChainError> {
         let mut pubkeys = vec![];
         let mut indices = HashMap::new();
+        let mut pubkey_bytes = vec![];
 
         for validator_index in 0.. {
             if let Some(DatabasePubkey(pubkey)) =
                 store.get_item(&DatabasePubkey::key_for_index(validator_index))?
             {
                 pubkeys.push((&pubkey).try_into().map_err(Error::PubkeyDecode)?);
+                pubkey_bytes.push(pubkey);
                 indices.insert(pubkey, validator_index);
             } else {
                 break;
@@ -73,6 +77,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         Ok(ValidatorPubkeyCache {
             pubkeys,
             indices,
+            pubkey_bytes,
             backing: PubkeyCacheBacking::Database(store),
         })
     }
@@ -91,6 +96,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         let mut result = ValidatorPubkeyCache {
             pubkeys: Vec::with_capacity(existing_cache.pubkeys.len()),
             indices: HashMap::with_capacity(existing_cache.indices.len()),
+            pubkey_bytes: Vec::with_capacity(existing_cache.indices.len()),
             backing: PubkeyCacheBacking::Database(store),
         };
         result.import(existing_cache.pubkeys.iter().map(PublicKeyBytes::from))?;
@@ -120,6 +126,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
     where
         I: Iterator<Item = PublicKeyBytes> + ExactSizeIterator,
     {
+        self.pubkey_bytes.reserve(validator_keys.len());
         self.pubkeys.reserve(validator_keys.len());
         self.indices.reserve(validator_keys.len());
 
@@ -153,6 +160,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
                     .try_into()
                     .map_err(BeaconChainError::InvalidValidatorPubkeyBytes)?,
             );
+            self.pubkey_bytes.push(pubkey);
 
             self.indices.insert(pubkey, i);
         }
@@ -165,6 +173,11 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         self.pubkeys.get(i)
     }
 
+    /// Get the public key (in bytes form) for a validator with index `i`.
+    pub fn get_pubkey_bytes(&self, i: usize) -> Option<&PublicKeyBytes> {
+        self.pubkey_bytes.get(i)
+    }
+
     /// Get the index of a validator with `pubkey`.
     pub fn get_index(&self, pubkey: &PublicKeyBytes) -> Option<usize> {
         self.indices.get(pubkey).copied()
@@ -264,13 +277,15 @@ impl ValidatorPubkeyCacheFile {
 
         let mut last = None;
         let mut pubkeys = Vec::with_capacity(list.len());
-        let mut indices = HashMap::new();
+        let mut indices = HashMap::with_capacity(list.len());
+        let mut pubkey_bytes = Vec::with_capacity(list.len());
 
         for (index, pubkey) in list {
             let expected = last.map(|n| n + 1);
             if expected.map_or(true, |expected| index == expected) {
                 last = Some(index);
                 pubkeys.push((&pubkey).try_into().map_err(Error::PubkeyDecode)?);
+                pubkey_bytes.push(pubkey);
                 indices.insert(pubkey, index);
             } else {
                 return Err(Error::InconsistentIndex {
@@ -283,6 +298,7 @@ impl ValidatorPubkeyCacheFile {
         Ok(ValidatorPubkeyCache {
             pubkeys,
             indices,
+            pubkey_bytes,
             backing: PubkeyCacheBacking::File(self),
         })
     }