State cache tweaks (#7095)

Backport of:

- https://github.com/sigp/lighthouse/pull/7067

For:

- https://github.com/sigp/lighthouse/issues/7039


  - Prevent writing to state cache when migrating the database
- Add `state-cache-headroom` flag to control pruning
- Prune old epoch boundary states ahead of mid-epoch states
- Never prune head block's state
- Avoid caching ancestor states unless they are on an epoch boundary
- Log when states enter/exit the cache

Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu>

beacon_node/beacon_chain/src/attestation_rewards.rs

@@ -47,8 +47,10 @@ impl<T: BeaconChainTypes> BeaconChain<T> {
             .state_root_at_slot(state_slot)?
             .ok_or(BeaconChainError::NoStateForSlot(state_slot))?;
 
+        // This branch is reached from the HTTP API. We assume the user wants
+        // to cache states so that future calls are faster.
         let state = self
-            .get_state(&state_root, Some(state_slot))?
+            .get_state(&state_root, Some(state_slot), true)?
             .ok_or(BeaconChainError::MissingBeaconState(state_root))?;
 
         if state.fork_name_unchecked().altair_enabled() {