Add configurable block replayer (#2863)

## Issue Addressed

Successor to #2431

## Proposed Changes

* Add a `BlockReplayer` struct to abstract over the intricacies of calling `per_slot_processing` and `per_block_processing` while avoiding unnecessary tree hashing.
* Add a variant of the forwards state root iterator that does not require an `end_state`.
* Use the `BlockReplayer` when reconstructing states in the database. Use the efficient forwards iterator for frozen states.
* Refactor the iterators to remove `Arc<HotColdDB>` (this seems to be neater than making _everything_ an `Arc<HotColdDB>` as I did in #2431).

Supplying the state roots allow us to avoid building a tree hash cache at all when reconstructing historic states, which saves around 1 second flat (regardless of `slots-per-restore-point`). This is a small percentage of worst-case state load times with 200K validators and SPRP=2048 (~15s vs ~16s) but a significant speed-up for more frequent restore points: state loads with SPRP=32 should be now consistently <500ms instead of 1.5s (a ~3x speedup).

## Additional Info

Required by https://github.com/sigp/lighthouse/pull/2628

testing/ef_tests/src/cases/operations.rs

@@ -12,7 +12,7 @@ use state_processing::per_block_processing::{
         altair, base, process_attester_slashings, process_deposits, process_exits,
         process_proposer_slashings,
     },
-    process_sync_aggregate, VerifySignatures,
+    process_sync_aggregate, VerifyBlockRoot, VerifySignatures,
 };
 use std::fmt::Debug;
 use std::path::Path;
@@ -183,7 +183,7 @@ impl<E: EthSpec> Operation<E> for BeaconBlock<E> {
         spec: &ChainSpec,
         _: &Operations<E, Self>,
     ) -> Result<(), BlockProcessingError> {
-        process_block_header(state, self.to_ref(), spec)?;
+        process_block_header(state, self.to_ref(), VerifyBlockRoot::True, spec)?;
         Ok(())
     }
 }

testing/ef_tests/src/cases/sanity_blocks.rs

@@ -5,6 +5,7 @@ use crate::decode::{ssz_decode_file_with, ssz_decode_state, yaml_decode_file};
 use serde_derive::Deserialize;
 use state_processing::{
     per_block_processing, per_slot_processing, BlockProcessingError, BlockSignatureStrategy,
+    VerifyBlockRoot,
 };
 use types::{BeaconState, EthSpec, ForkName, RelativeEpoch, SignedBeaconBlock};
 
@@ -98,6 +99,7 @@ impl<E: EthSpec> Case for SanityBlocks<E> {
                     signed_block,
                     None,
                     BlockSignatureStrategy::VerifyIndividual,
+                    VerifyBlockRoot::True,
                     spec,
                 )?;
 
@@ -106,6 +108,7 @@ impl<E: EthSpec> Case for SanityBlocks<E> {
                     signed_block,
                     None,
                     BlockSignatureStrategy::VerifyBulk,
+                    VerifyBlockRoot::True,
                     spec,
                 )?;
 

testing/ef_tests/src/cases/transition.rs

@@ -4,6 +4,7 @@ use crate::decode::{ssz_decode_file_with, ssz_decode_state, yaml_decode_file};
 use serde_derive::Deserialize;
 use state_processing::{
     per_block_processing, state_advance::complete_state_advance, BlockSignatureStrategy,
+    VerifyBlockRoot,
 };
 use std::str::FromStr;
 use types::{BeaconState, Epoch, ForkName, SignedBeaconBlock};
@@ -97,6 +98,7 @@ impl<E: EthSpec> Case for TransitionTest<E> {
                     block,
                     None,
                     BlockSignatureStrategy::VerifyBulk,
+                    VerifyBlockRoot::True,
                     spec,
                 )
                 .map_err(|e| format!("Block processing failed: {:?}", e))?;