Optimize validator duties (#2243)

## Issue Addressed

Closes #2052

## Proposed Changes

- Refactor the attester/proposer duties endpoints in the BN
    - Performance improvements
    - Fixes some potential inconsistencies with the dependent root fields.
    - Removes `http_api::beacon_proposer_cache` and just uses the one on the `BeaconChain` instead.
    - Move the code for the proposer/attester duties endpoints into separate files, for readability.
- Refactor the `DutiesService` in the VC
    - Required to reduce the delay on broadcasting new blocks.
    - Gets rid of the `ValidatorDuty` shim struct that came about when we adopted the standard API.
    - Separate block/attestation duty tasks so that they don't block each other when one is slow.
- In the VC, use `PublicKeyBytes` to represent validators instead of `PublicKey`. `PublicKey` is a legit crypto object whilst `PublicKeyBytes` is just a byte-array, it's much faster to clone/hash `PublicKeyBytes` and this change has had a significant impact on runtimes.
    - Unfortunately this has created lots of dust changes.
 - In the BN, store `PublicKeyBytes` in the `beacon_proposer_cache` and allow access to them. The HTTP API always sends `PublicKeyBytes` over the wire and the conversion from `PublicKey` -> `PublickeyBytes` is non-trivial, especially when queries have 100s/1000s of validators (like Pyrmont).
 - Add the `state_processing::state_advance` mod which dedups a lot of the "apply `n` skip slots to the state" code.
    - This also fixes a bug with some functions which were failing to include a state root as per [this comment](072695284f/consensus/state_processing/src/state_advance.rs (L69-L74)). I couldn't find any instance of this bug that resulted in anything more severe than keying a shuffling cache by the wrong block root.
 - Swap the VC block service to use `mpsc` from `tokio` instead of `futures`. This is consistent with the rest of the code base.
    
~~This PR *reduces* the size of the codebase 🎉~~ It *used* to reduce the size of the code base before I added more comments. 

## Observations on Prymont

- Proposer duties times down from peaks of 450ms to consistent <1ms.
- Current epoch attester duties times down from >1s peaks to a consistent 20-30ms.
- Block production down from +600ms to 100-200ms.

## Additional Info

- ~~Blocked on #2241~~
- ~~Blocked on #2234~~

## TODO

- [x] ~~Refactor this into some smaller PRs?~~ Leaving this as-is for now.
- [x] Address `per_slot_processing` roots.
- [x] Investigate slow next epoch times. Not getting added to cache on block processing?
- [x] Consider [this](072695284f/beacon_node/store/src/hot_cold_store.rs (L811-L812)) in the scenario of replacing the state roots


Co-authored-by: pawan <pawandhananjay@gmail.com>
Co-authored-by: Michael Sproul <michael@sigmaprime.io>

beacon_node/beacon_chain/src/attestation_verification.rs

@@ -1073,7 +1073,7 @@ where
     }
 
     chain
-        .with_committee_cache(target.root, attestation_epoch, |committee_cache| {
+        .with_committee_cache(target.root, attestation_epoch, |committee_cache, _| {
             let committees_per_slot = committee_cache.committees_per_slot();
 
             Ok(committee_cache

beacon_node/beacon_chain/src/beacon_chain.rs

@@ -42,8 +42,11 @@ use slasher::Slasher;
 use slog::{crit, debug, error, info, trace, warn, Logger};
 use slot_clock::SlotClock;
 use state_processing::{
-    common::get_indexed_attestation, per_block_processing,
-    per_block_processing::errors::AttestationValidationError, per_slot_processing,
+    common::get_indexed_attestation,
+    per_block_processing,
+    per_block_processing::errors::AttestationValidationError,
+    per_slot_processing,
+    state_advance::{complete_state_advance, partial_state_advance},
     BlockSignatureStrategy, SigVerifiedOp,
 };
 use std::borrow::Cow;
@@ -156,6 +159,7 @@ pub struct HeadInfo {
     pub fork: Fork,
     pub genesis_time: u64,
     pub genesis_validators_root: Hash256,
+    pub proposer_shuffling_decision_root: Hash256,
 }
 
 pub trait BeaconChainTypes: Send + Sync + 'static {
@@ -243,7 +247,7 @@ pub struct BeaconChain<T: BeaconChainTypes> {
     /// Caches the attester shuffling for a given epoch and shuffling key root.
     pub(crate) shuffling_cache: TimeoutRwLock<ShufflingCache>,
     /// Caches the beacon block proposer shuffling for a given epoch and shuffling key root.
-    pub(crate) beacon_proposer_cache: Mutex<BeaconProposerCache>,
+    pub beacon_proposer_cache: Mutex<BeaconProposerCache>,
     /// Caches a map of `validator_index -> validator_pubkey`.
     pub(crate) validator_pubkey_cache: TimeoutRwLock<ValidatorPubkeyCache<T>>,
     /// A list of any hard-coded forks that have been disabled.
@@ -606,6 +610,10 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     /// A summarized version of `Self::head` that involves less cloning.
     pub fn head_info(&self) -> Result<HeadInfo, Error> {
         self.with_head(|head| {
+            let proposer_shuffling_decision_root = head
+                .beacon_state
+                .proposer_shuffling_decision_root(head.beacon_block_root)?;
+
             Ok(HeadInfo {
                 slot: head.beacon_block.slot(),
                 block_root: head.beacon_block_root,
@@ -615,6 +623,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                 fork: head.beacon_state.fork,
                 genesis_time: head.beacon_state.genesis_time,
                 genesis_validators_root: head.beacon_state.genesis_validators_root,
+                proposer_shuffling_decision_root,
             })
         })
     }
@@ -773,6 +782,42 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         Ok(pubkey_cache.get(validator_index).cloned())
     }
 
+    /// As per `Self::validator_pubkey`, but returns `PublicKeyBytes`.
+    pub fn validator_pubkey_bytes(
+        &self,
+        validator_index: usize,
+    ) -> Result<Option<PublicKeyBytes>, Error> {
+        let pubkey_cache = self
+            .validator_pubkey_cache
+            .try_read_for(VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT)
+            .ok_or(Error::ValidatorPubkeyCacheLockTimeout)?;
+
+        Ok(pubkey_cache.get_pubkey_bytes(validator_index).copied())
+    }
+
+    /// As per `Self::validator_pubkey_bytes` but will resolve multiple indices at once to avoid
+    /// bouncing the read-lock on the pubkey cache.
+    ///
+    /// Returns a map that may have a length less than `validator_indices.len()` if some indices
+    /// were unable to be resolved.
+    pub fn validator_pubkey_bytes_many(
+        &self,
+        validator_indices: &[usize],
+    ) -> Result<HashMap<usize, PublicKeyBytes>, Error> {
+        let pubkey_cache = self
+            .validator_pubkey_cache
+            .try_read_for(VALIDATOR_PUBKEY_CACHE_LOCK_TIMEOUT)
+            .ok_or(Error::ValidatorPubkeyCacheLockTimeout)?;
+
+        let mut map = HashMap::with_capacity(validator_indices.len());
+        for &validator_index in validator_indices {
+            if let Some(pubkey) = pubkey_cache.get_pubkey_bytes(validator_index) {
+                map.insert(validator_index, *pubkey);
+            }
+        }
+        Ok(map)
+    }
+
     /// Returns the block canonical root of the current canonical chain at a given slot.
     ///
     /// Returns `None` if the given slot doesn't exist in the chain.
@@ -803,19 +848,35 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         })
     }
 
-    /// Returns the attestation duties for a given validator index.
+    /// Returns the attestation duties for the given validator indices using the shuffling cache.
     ///
-    /// Information is read from the current state, so only information from the present and prior
-    /// epoch is available.
-    pub fn validator_attestation_duty(
+    /// An error may be returned if `head_block_root` is a finalized block, this function is only
+    /// designed for operations at the head of the chain.
+    ///
+    /// The returned `Vec` will have the same length as `validator_indices`, any
+    /// non-existing/inactive validators will have `None` values.
+    ///
+    /// ## Notes
+    ///
+    /// This function will try to use the shuffling cache to return the value. If the value is not
+    /// in the shuffling cache, it will be added. Care should be taken not to wash out the
+    /// shuffling cache with historical/useless values.
+    pub fn validator_attestation_duties(
         &self,
-        validator_index: usize,
+        validator_indices: &[u64],
         epoch: Epoch,
-    ) -> Result<Option<AttestationDuty>, Error> {
-        let head_block_root = self.head_beacon_block_root()?;
+        head_block_root: Hash256,
+    ) -> Result<(Vec<Option<AttestationDuty>>, Hash256), Error> {
+        self.with_committee_cache(head_block_root, epoch, |committee_cache, dependent_root| {
+            let duties = validator_indices
+                .iter()
+                .map(|validator_index| {
+                    let validator_index = *validator_index as usize;
+                    committee_cache.get_attestation_duties(validator_index)
+                })
+                .collect();
 
-        self.with_committee_cache(head_block_root, epoch, |committee_cache| {
-            Ok(committee_cache.get_attestation_duties(validator_index))
+            Ok((duties, dependent_root))
         })
     }
 
@@ -867,6 +928,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
                 index,
                 head.beacon_block_root,
                 Cow::Borrowed(&head.beacon_state),
+                head.beacon_state_root(),
             )
         } else {
             // We disallow producing attestations *prior* to the current head since such an
@@ -902,6 +964,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         index: CommitteeIndex,
         beacon_block_root: Hash256,
         mut state: Cow<BeaconState<T::EthSpec>>,
+        state_root: Hash256,
     ) -> Result<Attestation<T::EthSpec>, Error> {
         let epoch = slot.epoch(T::EthSpec::slots_per_epoch());
 
@@ -909,13 +972,14 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             return Err(Error::CannotAttestToFutureState);
         } else if state.current_epoch() < epoch {
             let mut_state = state.to_mut();
-            while mut_state.current_epoch() < epoch {
-                // Note: here we provide `Hash256::zero()` as the root of the current state. This
-                // has the effect of setting the values of all historic state roots to the zero
-                // hash. This is an optimization, we don't need the state roots so why calculate
-                // them?
-                per_slot_processing(mut_state, Some(Hash256::zero()), &self.spec)?;
-            }
+            // Only perform a "partial" state advance since we do not require the state roots to be
+            // accurate.
+            partial_state_advance(
+                mut_state,
+                Some(state_root),
+                epoch.start_slot(T::EthSpec::slots_per_epoch()),
+                &self.spec,
+            )?;
             mut_state.build_committee_cache(RelativeEpoch::Current, &self.spec)?;
         }
 
@@ -1861,7 +1925,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     pub fn produce_block_on_state(
         &self,
         mut state: BeaconState<T::EthSpec>,
-        mut state_root_opt: Option<Hash256>,
+        state_root_opt: Option<Hash256>,
         produce_at_slot: Slot,
         randao_reveal: Signature,
         validator_graffiti: Option<Graffiti>,
@@ -1880,15 +1944,10 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         }
 
         let slot_timer = metrics::start_timer(&metrics::BLOCK_PRODUCTION_SLOT_PROCESS_TIMES);
-        // If required, transition the new state to the present slot.
-        //
-        // Note: supplying some `state_root` when it it is known would be a cheap and easy
-        // optimization.
-        while state.slot < produce_at_slot {
-            // Using `state_root.take()` here ensures that we consume the `state_root` on the first
-            // iteration and never use it again.
-            per_slot_processing(&mut state, state_root_opt.take(), &self.spec)?;
-        }
+
+        // Ensure the state has performed a complete transition into the required slot.
+        complete_state_advance(&mut state, state_root_opt, produce_at_slot, &self.spec)?;
+
         drop(slot_timer);
 
         state.build_committee_cache(RelativeEpoch::Current, &self.spec)?;
@@ -2363,7 +2422,10 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     }
 
     /// Runs the `map_fn` with the committee cache for `shuffling_epoch` from the chain with head
-    /// `head_block_root`.
+    /// `head_block_root`. The `map_fn` will be supplied two values:
+    ///
+    /// - `&CommitteeCache`: the committee cache that serves the given parameters.
+    /// - `Hash256`: the "shuffling decision root" which uniquely identifies the `CommitteeCache`.
     ///
     /// It's not necessary that `head_block_root` matches our current view of the chain, it can be
     /// any block that is:
@@ -2376,7 +2438,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
     ///
     /// ## Important
     ///
-    /// This function is **not** suitable for determining proposer duties.
+    /// This function is **not** suitable for determining proposer duties (only attester duties).
     ///
     /// ## Notes
     ///
@@ -2394,7 +2456,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         map_fn: F,
     ) -> Result<R, Error>
     where
-        F: Fn(&CommitteeCache) -> Result<R, Error>,
+        F: Fn(&CommitteeCache, Hash256) -> Result<R, Error>,
     {
         let head_block = self
             .fork_choice
@@ -2425,7 +2487,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
         metrics::stop_timer(cache_wait_timer);
 
         if let Some(committee_cache) = shuffling_cache.get(&shuffling_id) {
-            map_fn(committee_cache)
+            map_fn(committee_cache, shuffling_id.shuffling_decision_block)
         } else {
             // Drop the shuffling cache to avoid holding the lock for any longer than
             // required.
@@ -2434,33 +2496,80 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             debug!(
                 self.log,
                 "Committee cache miss";
-                "shuffling_epoch" => shuffling_epoch.as_u64(),
+                "shuffling_id" => ?shuffling_epoch,
                 "head_block_root" => head_block_root.to_string(),
             );
 
             let state_read_timer =
                 metrics::start_timer(&metrics::ATTESTATION_PROCESSING_STATE_READ_TIMES);
 
-            let mut state = self
-                .store
-                .get_inconsistent_state_for_attestation_verification_only(
-                    &head_block.state_root,
-                    Some(head_block.slot),
-                )?
-                .ok_or(Error::MissingBeaconState(head_block.state_root))?;
+            // If the head of the chain can serve this request, use it.
+            //
+            // This code is a little awkward because we need to ensure that the head we read and
+            // the head we copy is identical. Taking one lock to read the head values and another
+            // to copy the head is liable to race-conditions.
+            let head_state_opt = self.with_head(|head| {
+                if head.beacon_block_root == head_block_root {
+                    Ok(Some((
+                        head.beacon_state
+                            .clone_with(CloneConfig::committee_caches_only()),
+                        head.beacon_state_root(),
+                    )))
+                } else {
+                    Ok::<_, Error>(None)
+                }
+            })?;
+
+            // If the head state is useful for this request, use it. Otherwise, read a state from
+            // disk.
+            let (mut state, state_root) = if let Some((state, state_root)) = head_state_opt {
+                (state, state_root)
+            } else {
+                let state_root = head_block.state_root;
+                let state = self
+                    .store
+                    .get_inconsistent_state_for_attestation_verification_only(
+                        &state_root,
+                        Some(head_block.slot),
+                    )?
+                    .ok_or(Error::MissingBeaconState(head_block.state_root))?;
+                (state, state_root)
+            };
+
+            /*
+             * IMPORTANT
+             *
+             * Since it's possible that
+             * `Store::get_inconsistent_state_for_attestation_verification_only` was used to obtain
+             * the state, we cannot rely upon the following fields:
+             *
+             * - `state.state_roots`
+             * - `state.block_roots`
+             *
+             * These fields should not be used for the rest of this function.
+             */
 
             metrics::stop_timer(state_read_timer);
             let state_skip_timer =
                 metrics::start_timer(&metrics::ATTESTATION_PROCESSING_STATE_SKIP_TIMES);
 
-            while state.current_epoch() + 1 < shuffling_epoch {
-                // Here we tell `per_slot_processing` to skip hashing the state and just
-                // use the zero hash instead.
-                //
-                // The state roots are not useful for the shuffling, so there's no need to
-                // compute them.
-                per_slot_processing(&mut state, Some(Hash256::zero()), &self.spec)
-                    .map_err(Error::from)?;
+            // If the state is in an earlier epoch, advance it. If it's from a later epoch, reject
+            // it.
+            if state.current_epoch() + 1 < shuffling_epoch {
+                // Since there's a one-epoch look-ahead on the attester shuffling, it suffices to
+                // only advance into the slot prior to the `shuffling_epoch`.
+                let target_slot = shuffling_epoch
+                    .saturating_sub(1_u64)
+                    .start_slot(T::EthSpec::slots_per_epoch());
+
+                // Advance the state into the required slot, using the "partial" method since the state
+                // roots are not relevant for the shuffling.
+                partial_state_advance(&mut state, Some(state_root), target_slot, &self.spec)?;
+            } else if state.current_epoch() > shuffling_epoch {
+                return Err(Error::InvalidStateForShuffling {
+                    state_epoch: state.current_epoch(),
+                    shuffling_epoch,
+                });
             }
 
             metrics::stop_timer(state_skip_timer);
@@ -2473,6 +2582,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             state.build_committee_cache(relative_epoch, &self.spec)?;
 
             let committee_cache = state.committee_cache(relative_epoch)?;
+            let shuffling_decision_block = shuffling_id.shuffling_decision_block;
 
             self.shuffling_cache
                 .try_write_for(ATTESTATION_CACHE_LOCK_TIMEOUT)
@@ -2481,7 +2591,7 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
 
             metrics::stop_timer(committee_building_timer);
 
-            map_fn(&committee_cache)
+            map_fn(&committee_cache, shuffling_decision_block)
         }
     }
 

beacon_node/beacon_chain/src/beacon_proposer_cache.rs

@@ -59,7 +59,7 @@ impl Default for BeaconProposerCache {
 impl BeaconProposerCache {
     /// If it is cached, returns the proposer for the block at `slot` where the block has the
     /// ancestor block root of `shuffling_decision_block` at `end_slot(slot.epoch() - 1)`.
-    pub fn get<T: EthSpec>(
+    pub fn get_slot<T: EthSpec>(
         &mut self,
         shuffling_decision_block: Hash256,
         slot: Slot,
@@ -84,6 +84,20 @@ impl BeaconProposerCache {
         }
     }
 
+    /// As per `Self::get_slot`, but returns all proposers in all slots for the given `epoch`.
+    ///
+    /// The nth slot in the returned `SmallVec` will be equal to the nth slot in the given `epoch`.
+    /// E.g., if `epoch == 1` then `smallvec[0]` refers to slot 32 (assuming `SLOTS_PER_EPOCH ==
+    /// 32`).
+    pub fn get_epoch<T: EthSpec>(
+        &mut self,
+        shuffling_decision_block: Hash256,
+        epoch: Epoch,
+    ) -> Option<&SmallVec<[usize; TYPICAL_SLOTS_PER_EPOCH]>> {
+        let key = (epoch, shuffling_decision_block);
+        self.cache.get(&key).map(|cache| &cache.proposers)
+    }
+
     /// Insert the proposers into the cache.
     ///
     /// See `Self::get` for a description of `shuffling_decision_block`.

beacon_node/beacon_chain/src/block_verification.rs

@@ -60,7 +60,9 @@ use state_processing::{
     block_signature_verifier::{BlockSignatureVerifier, Error as BlockSignatureVerifierError},
     per_block_processing,
     per_epoch_processing::EpochProcessingSummary,
-    per_slot_processing, BlockProcessingError, BlockSignatureStrategy, SlotProcessingError,
+    per_slot_processing,
+    state_advance::partial_state_advance,
+    BlockProcessingError, BlockSignatureStrategy, SlotProcessingError,
 };
 use std::borrow::Cow;
 use std::convert::TryFrom;
@@ -351,8 +353,12 @@ pub fn signature_verify_chain_segment<T: BeaconChainTypes>(
         .map(|(_, block)| block.slot())
         .unwrap_or_else(|| slot);
 
-    let state =
-        cheap_state_advance_to_obtain_committees(&mut parent.pre_state, highest_slot, &chain.spec)?;
+    let state = cheap_state_advance_to_obtain_committees(
+        &mut parent.pre_state,
+        parent.beacon_state_root,
+        highest_slot,
+        &chain.spec,
+    )?;
 
     let pubkey_cache = get_validator_pubkey_cache(chain)?;
     let mut signature_verifier = get_signature_verifier(&state, &pubkey_cache, &chain.spec);
@@ -564,7 +570,7 @@ impl<T: BeaconChainTypes> GossipVerifiedBlock<T> {
         let proposer_opt = chain
             .beacon_proposer_cache
             .lock()
-            .get::<T::EthSpec>(proposer_shuffling_decision_block, block.slot());
+            .get_slot::<T::EthSpec>(proposer_shuffling_decision_block, block.slot());
         let (expected_proposer, fork, parent, block) = if let Some(proposer) = proposer_opt {
             // The proposer index was cached and we can return it without needing to load the
             // parent.
@@ -586,6 +592,7 @@ impl<T: BeaconChainTypes> GossipVerifiedBlock<T> {
             // The state produced is only valid for determining proposer/attester shuffling indices.
             let state = cheap_state_advance_to_obtain_committees(
                 &mut parent.pre_state,
+                parent.beacon_state_root,
                 block.slot(),
                 &chain.spec,
             )?;
@@ -694,6 +701,7 @@ impl<T: BeaconChainTypes> SignatureVerifiedBlock<T> {
 
         let state = cheap_state_advance_to_obtain_committees(
             &mut parent.pre_state,
+            parent.beacon_state_root,
             block.slot(),
             &chain.spec,
         )?;
@@ -738,6 +746,7 @@ impl<T: BeaconChainTypes> SignatureVerifiedBlock<T> {
 
         let state = cheap_state_advance_to_obtain_committees(
             &mut parent.pre_state,
+            parent.beacon_state_root,
             block.slot(),
             &chain.spec,
         )?;
@@ -1280,6 +1289,7 @@ fn load_parent<T: BeaconChainTypes>(
                 beacon_block: parent_block,
                 beacon_block_root: root,
                 pre_state: parent_state,
+                beacon_state_root: Some(parent_state_root),
             },
             block,
         ))
@@ -1303,6 +1313,7 @@ fn load_parent<T: BeaconChainTypes>(
 /// mutated to be invalid (in fact, it is never changed beyond a simple committee cache build).
 fn cheap_state_advance_to_obtain_committees<'a, E: EthSpec>(
     state: &'a mut BeaconState<E>,
+    state_root_opt: Option<Hash256>,
     block_slot: Slot,
     spec: &ChainSpec,
 ) -> Result<Cow<'a, BeaconState<E>>, BlockError<E>> {
@@ -1319,14 +1330,12 @@ fn cheap_state_advance_to_obtain_committees<'a, E: EthSpec>(
         })
     } else {
         let mut state = state.clone_with(CloneConfig::committee_caches_only());
+        let target_slot = block_epoch.start_slot(E::slots_per_epoch());
 
-        while state.current_epoch() < block_epoch {
-            // Don't calculate state roots since they aren't required for calculating
-            // shuffling (achieved by providing Hash256::zero()).
-            per_slot_processing(&mut state, Some(Hash256::zero()), spec).map_err(|e| {
-                BlockError::BeaconChainError(BeaconChainError::SlotProcessingError(e))
-            })?;
-        }
+        // Advance the state into the same epoch as the block. Use the "partial" method since state
+        // roots are not important for proposer/attester shuffling.
+        partial_state_advance(&mut state, state_root_opt, target_slot, spec)
+            .map_err(|e| BlockError::BeaconChainError(BeaconChainError::from(e)))?;
 
         state.build_committee_cache(RelativeEpoch::Current, spec)?;
 

beacon_node/beacon_chain/src/errors.rs

@@ -17,6 +17,7 @@ use state_processing::{
         ProposerSlashingValidationError,
     },
     signature_sets::Error as SignatureSetError,
+    state_advance::Error as StateAdvanceError,
     BlockProcessingError, SlotProcessingError,
 };
 use std::time::Duration;
@@ -51,6 +52,7 @@ pub enum BeaconChainError {
     MissingBeaconBlock(Hash256),
     MissingBeaconState(Hash256),
     SlotProcessingError(SlotProcessingError),
+    StateAdvanceError(StateAdvanceError),
     UnableToAdvanceState(String),
     NoStateForAttestation {
         beacon_block_root: Hash256,
@@ -81,6 +83,7 @@ pub enum BeaconChainError {
     BlockSignatureVerifierError(state_processing::block_signature_verifier::Error),
     DuplicateValidatorPublicKey,
     ValidatorPubkeyCacheFileError(String),
+    ValidatorIndexUnknown(usize),
     OpPoolError(OpPoolError),
     NaiveAggregationError(NaiveAggregationError),
     ObservedAttestationsError(ObservedAttestationsError),
@@ -105,6 +108,10 @@ pub enum BeaconChainError {
         block_slot: Slot,
         state_slot: Slot,
     },
+    InvalidStateForShuffling {
+        state_epoch: Epoch,
+        shuffling_epoch: Epoch,
+    },
 }
 
 easy_from_to!(SlotProcessingError, BeaconChainError);
@@ -122,6 +129,7 @@ easy_from_to!(BlockSignatureVerifierError, BeaconChainError);
 easy_from_to!(PruningError, BeaconChainError);
 easy_from_to!(ArithError, BeaconChainError);
 easy_from_to!(ForkChoiceStoreError, BeaconChainError);
+easy_from_to!(StateAdvanceError, BeaconChainError);
 
 #[derive(Debug)]
 pub enum BlockProductionError {
@@ -133,6 +141,7 @@ pub enum BlockProductionError {
     BlockProcessingError(BlockProcessingError),
     Eth1ChainError(Eth1ChainError),
     BeaconStateError(BeaconStateError),
+    StateAdvanceError(StateAdvanceError),
     OpPoolError(OpPoolError),
     /// The `BeaconChain` was explicitly configured _without_ a connection to eth1, therefore it
     /// cannot produce blocks.
@@ -147,3 +156,4 @@ easy_from_to!(BlockProcessingError, BlockProductionError);
 easy_from_to!(BeaconStateError, BlockProductionError);
 easy_from_to!(SlotProcessingError, BlockProductionError);
 easy_from_to!(Eth1ChainError, BlockProductionError);
+easy_from_to!(StateAdvanceError, BlockProductionError);

beacon_node/beacon_chain/src/snapshot_cache.rs

@@ -14,14 +14,18 @@ pub struct PreProcessingSnapshot<T: EthSpec> {
     /// advanced forward one slot using `per_slot_processing`. This state is "primed and ready" for
     /// the application of another block.
     pub pre_state: BeaconState<T>,
+    /// This value is only set to `Some` if the `pre_state` was *not* advanced forward.
+    pub beacon_state_root: Option<Hash256>,
     pub beacon_block: SignedBeaconBlock<T>,
     pub beacon_block_root: Hash256,
 }
 
 impl<T: EthSpec> From<BeaconSnapshot<T>> for PreProcessingSnapshot<T> {
     fn from(snapshot: BeaconSnapshot<T>) -> Self {
+        let beacon_state_root = Some(snapshot.beacon_state_root());
         Self {
             pre_state: snapshot.beacon_state,
+            beacon_state_root,
             beacon_block: snapshot.beacon_block,
             beacon_block_root: snapshot.beacon_block_root,
         }
@@ -47,10 +51,15 @@ impl<T: EthSpec> CacheItem<T> {
     }
 
     pub fn into_pre_state(self) -> PreProcessingSnapshot<T> {
+        // Do not include the beacon state root if the state has been advanced.
+        let beacon_state_root =
+            Some(self.beacon_block.state_root()).filter(|_| self.pre_state.is_none());
+
         PreProcessingSnapshot {
             beacon_block: self.beacon_block,
             beacon_block_root: self.beacon_block_root,
             pre_state: self.pre_state.unwrap_or(self.beacon_state),
+            beacon_state_root,
         }
     }
 }

beacon_node/beacon_chain/src/state_advance_timer.rs

@@ -15,8 +15,9 @@
 //! 2. There's a possibility that the head block is never built upon, causing wasted CPU cycles.
 use crate::validator_monitor::HISTORIC_EPOCHS as VALIDATOR_MONITOR_HISTORIC_EPOCHS;
 use crate::{
-    beacon_chain::BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT, snapshot_cache::StateAdvance, BeaconChain,
-    BeaconChainError, BeaconChainTypes,
+    beacon_chain::{ATTESTATION_CACHE_LOCK_TIMEOUT, BLOCK_PROCESSING_CACHE_LOCK_TIMEOUT},
+    snapshot_cache::StateAdvance,
+    BeaconChain, BeaconChainError, BeaconChainTypes,
 };
 use slog::{debug, error, warn, Logger};
 use slot_clock::SlotClock;
@@ -27,7 +28,7 @@ use std::sync::{
 };
 use task_executor::TaskExecutor;
 use tokio::time::sleep;
-use types::{EthSpec, Hash256, Slot};
+use types::{AttestationShufflingId, EthSpec, Hash256, RelativeEpoch, Slot};
 
 /// If the head slot is more than `MAX_ADVANCE_DISTANCE` from the current slot, then don't perform
 /// the state advancement.
@@ -252,16 +253,22 @@ fn advance_head<T: BeaconChainTypes>(
         "current_slot" => current_slot,
     );
 
-    // If the advanced state is in a later epoch than where it started, pre-emptively add the
-    // proposer shuffling for the new epoch into the cache.
-    if state.current_epoch() > initial_epoch {
-        debug!(
-            log,
-            "Priming proposer cache";
-            "head_root" => ?head_root,
-            "state_epoch" => state.current_epoch(),
-            "current_epoch" => current_slot.epoch(T::EthSpec::slots_per_epoch()),
-        );
+    // Build the current epoch cache, to prepare to compute proposer duties.
+    state
+        .build_committee_cache(RelativeEpoch::Current, &beacon_chain.spec)
+        .map_err(BeaconChainError::from)?;
+    // Build the next epoch cache, to prepare to compute attester duties.
+    state
+        .build_committee_cache(RelativeEpoch::Next, &beacon_chain.spec)
+        .map_err(BeaconChainError::from)?;
+
+    // If the `pre_state` is in a later epoch than `state`, pre-emptively add the proposer shuffling
+    // for the state's current epoch and the committee cache for the state's next epoch.
+    if initial_epoch < state.current_epoch() {
+        // Update the proposer cache.
+        //
+        // We supply the `head_root` as the decision block since the prior `if` statement guarantees
+        // the head root is the latest block from the prior epoch.
         beacon_chain
             .beacon_proposer_cache
             .lock()
@@ -274,6 +281,27 @@ fn advance_head<T: BeaconChainTypes>(
                 state.fork,
             )
             .map_err(BeaconChainError::from)?;
+
+        // Update the attester cache.
+        let shuffling_id = AttestationShufflingId::new(head_root, &state, RelativeEpoch::Next)
+            .map_err(BeaconChainError::from)?;
+        let committee_cache = state
+            .committee_cache(RelativeEpoch::Next)
+            .map_err(BeaconChainError::from)?;
+        beacon_chain
+            .shuffling_cache
+            .try_write_for(ATTESTATION_CACHE_LOCK_TIMEOUT)
+            .ok_or(BeaconChainError::AttestationCacheLockTimeout)?
+            .insert(shuffling_id.clone(), committee_cache);
+
+        debug!(
+            log,
+            "Primed proposer and attester caches";
+            "head_root" => ?head_root,
+            "next_epoch_shuffling_root" => ?shuffling_id.shuffling_decision_block,
+            "state_epoch" => state.current_epoch(),
+            "current_epoch" => current_slot.epoch(T::EthSpec::slots_per_epoch()),
+        );
     }
 
     let final_slot = state.slot;

beacon_node/beacon_chain/src/test_utils.rs

@@ -19,7 +19,7 @@ use rand::SeedableRng;
 use rayon::prelude::*;
 use slog::Logger;
 use slot_clock::TestingSlotClock;
-use state_processing::per_slot_processing;
+use state_processing::state_advance::complete_state_advance;
 use std::borrow::Cow;
 use std::collections::{HashMap, HashSet};
 use std::sync::Arc;
@@ -331,6 +331,12 @@ where
         self.chain.head().unwrap().beacon_state
     }
 
+    pub fn get_current_state_and_root(&self) -> (BeaconState<E>, Hash256) {
+        let head = self.chain.head().unwrap();
+        let state_root = head.beacon_state_root();
+        (head.beacon_state, state_root)
+    }
+
     pub fn get_current_slot(&self) -> Slot {
         self.chain.slot().unwrap()
     }
@@ -377,10 +383,8 @@ where
         assert_ne!(slot, 0, "can't produce a block at slot 0");
         assert!(slot >= state.slot);
 
-        while state.slot < slot {
-            per_slot_processing(&mut state, None, &self.spec)
-                .expect("should be able to advance state to slot");
-        }
+        complete_state_advance(&mut state, None, slot, &self.spec)
+            .expect("should be able to advance state to slot");
 
         state
             .build_all_caches(&self.spec)
@@ -430,6 +434,7 @@ where
         &self,
         attesting_validators: &[usize],
         state: &BeaconState<E>,
+        state_root: Hash256,
         head_block_root: SignedBeaconBlockHash,
         attestation_slot: Slot,
     ) -> Vec<Vec<(Attestation<E>, SubnetId)>> {
@@ -454,6 +459,7 @@ where
                                 bc.index,
                                 head_block_root.into(),
                                 Cow::Borrowed(state),
+                                state_root,
                             )
                             .unwrap();
 
@@ -503,6 +509,7 @@ where
         &self,
         attestation_strategy: &AttestationStrategy,
         state: &BeaconState<E>,
+        state_root: Hash256,
         head_block_root: Hash256,
         attestation_slot: Slot,
     ) -> Vec<Vec<(Attestation<E>, SubnetId)>> {
@@ -513,6 +520,7 @@ where
         self.make_unaggregated_attestations(
             &validators,
             state,
+            state_root,
             head_block_root.into(),
             attestation_slot,
         )
@@ -522,11 +530,17 @@ where
         &self,
         attesting_validators: &[usize],
         state: &BeaconState<E>,
+        state_root: Hash256,
         block_hash: SignedBeaconBlockHash,
         slot: Slot,
     ) -> HarnessAttestations<E> {
-        let unaggregated_attestations =
-            self.make_unaggregated_attestations(&attesting_validators, &state, block_hash, slot);
+        let unaggregated_attestations = self.make_unaggregated_attestations(
+            &attesting_validators,
+            &state,
+            state_root,
+            block_hash,
+            slot,
+        );
 
         let aggregated_attestations: Vec<Option<SignedAggregateAndProof<E>>> = unaggregated_attestations
             .iter()
@@ -754,12 +768,18 @@ where
     pub fn attest_block(
         &self,
         state: &BeaconState<E>,
+        state_root: Hash256,
         block_hash: SignedBeaconBlockHash,
         block: &SignedBeaconBlock<E>,
         validators: &[usize],
     ) {
-        let attestations =
-            self.make_attestations(validators, &state, block_hash, block.message.slot);
+        let attestations = self.make_attestations(
+            validators,
+            &state,
+            state_root,
+            block_hash,
+            block.message.slot,
+        );
         self.process_attestations(attestations);
     }
 
@@ -767,26 +787,29 @@ where
         &self,
         slot: Slot,
         state: BeaconState<E>,
+        state_root: Hash256,
         validators: &[usize],
     ) -> Result<(SignedBeaconBlockHash, BeaconState<E>), BlockError<E>> {
         let (block_hash, block, state) = self.add_block_at_slot(slot, state)?;
-        self.attest_block(&state, block_hash, &block, validators);
+        self.attest_block(&state, state_root, block_hash, &block, validators);
         Ok((block_hash, state))
     }
 
     pub fn add_attested_blocks_at_slots(
         &self,
         state: BeaconState<E>,
+        state_root: Hash256,
         slots: &[Slot],
         validators: &[usize],
     ) -> AddBlocksResult<E> {
         assert!(!slots.is_empty());
-        self.add_attested_blocks_at_slots_given_lbh(state, slots, validators, None)
+        self.add_attested_blocks_at_slots_given_lbh(state, state_root, slots, validators, None)
     }
 
     fn add_attested_blocks_at_slots_given_lbh(
         &self,
         mut state: BeaconState<E>,
+        state_root: Hash256,
         slots: &[Slot],
         validators: &[usize],
         mut latest_block_hash: Option<SignedBeaconBlockHash>,
@@ -799,7 +822,7 @@ where
         let mut state_hash_from_slot: HashMap<Slot, BeaconStateHash> = HashMap::new();
         for slot in slots {
             let (block_hash, new_state) = self
-                .add_attested_block_at_slot(*slot, state, validators)
+                .add_attested_block_at_slot(*slot, state, state_root, validators)
                 .unwrap();
             state = new_state;
             block_hash_from_slot.insert(*slot, block_hash);
@@ -857,8 +880,14 @@ where
         for epoch in min_epoch.as_u64()..=max_epoch.as_u64() {
             let mut new_chains = vec![];
 
-            for (head_state, slots, validators, mut block_hashes, mut state_hashes, head_block) in
-                chains
+            for (
+                mut head_state,
+                slots,
+                validators,
+                mut block_hashes,
+                mut state_hashes,
+                head_block,
+            ) in chains
             {
                 let epoch_slots = slots
                     .iter()
@@ -866,9 +895,11 @@ where
                     .copied()
                     .collect::<Vec<_>>();
 
+                let head_state_root = head_state.update_tree_hash_cache().unwrap();
                 let (new_block_hashes, new_state_hashes, new_head_block, new_head_state) = self
                     .add_attested_blocks_at_slots_given_lbh(
                         head_state,
+                        head_state_root,
                         &epoch_slots,
                         &validators,
                         Some(head_block),
@@ -947,7 +978,7 @@ where
         block_strategy: BlockStrategy,
         attestation_strategy: AttestationStrategy,
     ) -> Hash256 {
-        let (state, slots) = match block_strategy {
+        let (mut state, slots) = match block_strategy {
             BlockStrategy::OnCanonicalHead => {
                 let current_slot: u64 = self.get_current_slot().into();
                 let slots: Vec<Slot> = (current_slot..(current_slot + (num_blocks as u64)))
@@ -975,8 +1006,9 @@ where
             AttestationStrategy::AllValidators => self.get_all_validators(),
             AttestationStrategy::SomeValidators(vals) => vals,
         };
+        let state_root = state.update_tree_hash_cache().unwrap();
         let (_, _, last_produced_block_hash, _) =
-            self.add_attested_blocks_at_slots(state, &slots, &validators);
+            self.add_attested_blocks_at_slots(state, state_root, &slots, &validators);
         last_produced_block_hash.into()
     }
 

beacon_node/beacon_chain/src/validator_pubkey_cache.rs

@@ -23,6 +23,7 @@ use types::{BeaconState, Hash256, PublicKey, PublicKeyBytes};
 pub struct ValidatorPubkeyCache<T: BeaconChainTypes> {
     pubkeys: Vec<PublicKey>,
     indices: HashMap<PublicKeyBytes, usize>,
+    pubkey_bytes: Vec<PublicKeyBytes>,
     backing: PubkeyCacheBacking<T>,
 }
 
@@ -46,6 +47,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         let mut cache = Self {
             pubkeys: vec![],
             indices: HashMap::new(),
+            pubkey_bytes: vec![],
             backing: PubkeyCacheBacking::Database(store),
         };
 
@@ -58,12 +60,14 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
     pub fn load_from_store(store: BeaconStore<T>) -> Result<Self, BeaconChainError> {
         let mut pubkeys = vec![];
         let mut indices = HashMap::new();
+        let mut pubkey_bytes = vec![];
 
         for validator_index in 0.. {
             if let Some(DatabasePubkey(pubkey)) =
                 store.get_item(&DatabasePubkey::key_for_index(validator_index))?
             {
                 pubkeys.push((&pubkey).try_into().map_err(Error::PubkeyDecode)?);
+                pubkey_bytes.push(pubkey);
                 indices.insert(pubkey, validator_index);
             } else {
                 break;
@@ -73,6 +77,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         Ok(ValidatorPubkeyCache {
             pubkeys,
             indices,
+            pubkey_bytes,
             backing: PubkeyCacheBacking::Database(store),
         })
     }
@@ -91,6 +96,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         let mut result = ValidatorPubkeyCache {
             pubkeys: Vec::with_capacity(existing_cache.pubkeys.len()),
             indices: HashMap::with_capacity(existing_cache.indices.len()),
+            pubkey_bytes: Vec::with_capacity(existing_cache.indices.len()),
             backing: PubkeyCacheBacking::Database(store),
         };
         result.import(existing_cache.pubkeys.iter().map(PublicKeyBytes::from))?;
@@ -120,6 +126,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
     where
         I: Iterator<Item = PublicKeyBytes> + ExactSizeIterator,
     {
+        self.pubkey_bytes.reserve(validator_keys.len());
         self.pubkeys.reserve(validator_keys.len());
         self.indices.reserve(validator_keys.len());
 
@@ -153,6 +160,7 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
                     .try_into()
                     .map_err(BeaconChainError::InvalidValidatorPubkeyBytes)?,
             );
+            self.pubkey_bytes.push(pubkey);
 
             self.indices.insert(pubkey, i);
         }
@@ -165,6 +173,11 @@ impl<T: BeaconChainTypes> ValidatorPubkeyCache<T> {
         self.pubkeys.get(i)
     }
 
+    /// Get the public key (in bytes form) for a validator with index `i`.
+    pub fn get_pubkey_bytes(&self, i: usize) -> Option<&PublicKeyBytes> {
+        self.pubkey_bytes.get(i)
+    }
+
     /// Get the index of a validator with `pubkey`.
     pub fn get_index(&self, pubkey: &PublicKeyBytes) -> Option<usize> {
         self.indices.get(pubkey).copied()
@@ -264,13 +277,15 @@ impl ValidatorPubkeyCacheFile {
 
         let mut last = None;
         let mut pubkeys = Vec::with_capacity(list.len());
-        let mut indices = HashMap::new();
+        let mut indices = HashMap::with_capacity(list.len());
+        let mut pubkey_bytes = Vec::with_capacity(list.len());
 
         for (index, pubkey) in list {
             let expected = last.map(|n| n + 1);
             if expected.map_or(true, |expected| index == expected) {
                 last = Some(index);
                 pubkeys.push((&pubkey).try_into().map_err(Error::PubkeyDecode)?);
+                pubkey_bytes.push(pubkey);
                 indices.insert(pubkey, index);
             } else {
                 return Err(Error::InconsistentIndex {
@@ -283,6 +298,7 @@ impl ValidatorPubkeyCacheFile {
         Ok(ValidatorPubkeyCache {
             pubkeys,
             indices,
+            pubkey_bytes,
             backing: PubkeyCacheBacking::File(self),
         })
     }