State cache tweaks (#7095)

Backport of:

- https://github.com/sigp/lighthouse/pull/7067

For:

- https://github.com/sigp/lighthouse/issues/7039


  - Prevent writing to state cache when migrating the database
- Add `state-cache-headroom` flag to control pruning
- Prune old epoch boundary states ahead of mid-epoch states
- Never prune head block's state
- Avoid caching ancestor states unless they are on an epoch boundary
- Log when states enter/exit the cache

Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu>

beacon_node/beacon_chain/src/fork_revert.rs

@@ -116,8 +116,9 @@ pub fn reset_fork_choice_to_finalization<E: EthSpec, Hot: ItemStore<E>, Cold: It
 
     // Advance finalized state to finalized epoch (to handle skipped slots).
     let finalized_state_root = finalized_block.state_root();
+    // The enshrined finalized state should be in the state cache.
     let mut finalized_state = store
-        .get_state(&finalized_state_root, Some(finalized_block.slot()))
+        .get_state(&finalized_state_root, Some(finalized_block.slot()), true)
         .map_err(|e| format!("Error loading finalized state: {:?}", e))?
         .ok_or_else(|| {
             format!(