State cache tweaks (#7095)

Backport of:

- https://github.com/sigp/lighthouse/pull/7067

For:

- https://github.com/sigp/lighthouse/issues/7039


  - Prevent writing to state cache when migrating the database
- Add `state-cache-headroom` flag to control pruning
- Prune old epoch boundary states ahead of mid-epoch states
- Never prune head block's state
- Avoid caching ancestor states unless they are on an epoch boundary
- Log when states enter/exit the cache

Co-authored-by: Eitan Seri-Levi <eserilev@ucsc.edu>

beacon_node/beacon_chain/src/light_client_server_cache.rs

@@ -319,8 +319,11 @@ impl<T: BeaconChainTypes> LightClientServerCache<T> {
         metrics::inc_counter(&metrics::LIGHT_CLIENT_SERVER_CACHE_PREV_BLOCK_CACHE_MISS);
 
         // Compute the value, handling potential errors.
+        // This state should already be cached. By electing not to cache it here
+        // we remove any chance of the light client server from affecting the state cache.
+        // We'd like the light client server to be as minimally invasive as possible.
         let mut state = store
-            .get_state(block_state_root, Some(block_slot))?
+            .get_state(block_state_root, Some(block_slot), false)?
             .ok_or_else(|| {
                 BeaconChainError::DBInconsistent(format!("Missing state {:?}", block_state_root))
             })?;