Delete bogus InvalidBestNode error (#9364)

On Glamsterdam devnets we started seeing Lighthouse nodes unable to start with errors like:

> May 26 04:34:01.582 CRIT  Failed to start beacon node                   reason: "Unable to load fork choice from disk: ForkChoiceError(ProtoArrayStringError(\"find_head failed: InvalidBestNode(InvalidBestNodeInfo { current_slot: Slot(23550), start_root: 0x2c70b1641c29ec46360c99f9a8512f077862cbbc603e16f4a423007d210b0c5f, justified_checkpoint: Checkpoint { epoch: Epoch(712), root: 0x2c70b1641c29ec46360c99f9a8512f077862cbbc603e16f4a423007d210b0c5f }, finalized_checkpoint: Checkpoint { epoch: Epoch(710), root: 0xede5e0b09b51bdb5445ade3398e685bd193b845e0b0ffb827f0c3fec8277ea51 }, head_root: 0x2c70b1641c29ec46360c99f9a8512f077862cbbc603e16f4a423007d210b0c5f, head_justified_checkpoint: Checkpoint { epoch: Epoch(710), root: 0xede5e0b09b51bdb5445ade3398e685bd193b845e0b0ffb827f0c3fec8277ea51 }, head_finalized_checkpoint: Checkpoint { epoch: Epoch(709), root: 0xbb243eff616ff362c52b83113e7c536d0a68cb9ca3d6a1cb1055e732219d9736 } })\"))"

This error was the result of an overly-strict sanity check, based on assumptions that are not true under extreme network conditions.


  Completely remove the `InvalidBestNode` failure path: it is not compliant with the spec, and is actively harmful when triggered (it prevents Lighthouse from starting at all). The error was reachable in any situation where all leaf nodes of fork choice were ineligible to be the head. The payload invalidation tests show some examples of cases where this would happen, and the [newly-added regression test](9a5df1d982) shows a contrived case where it can happen on a Gloas network without _any_ slashings or invalid blocks. There are probably many more cases where it can happen.

We do not lose anything by removing it. The spec's implementation of `get_head` _always_ returns something (unless it crashes), and in these cases it is correct to return the starting node of the traversal: the justified checkpoint block. This is what we now do, and what the new test verifies.

I've also added some facilities to the harness for injecting attestations with fixed `payload_present` fields. @hopinheimer found himself needing something similar when messing with reorg tests, so I think these are probably useful. It might be possible to do without them by juggling the payload reveal timing in just the right way, but I think this approach is just way simpler.


Co-Authored-By: Michael Sproul <michael@sigmaprime.io>

consensus/proto_array/src/error.rs

@@ -1,6 +1,6 @@
 use crate::PayloadStatus;
 use safe_arith::ArithError;
-use types::{Checkpoint, Epoch, ExecutionBlockHash, Hash256, Slot};
+use types::{Epoch, ExecutionBlockHash, Hash256};
 
 #[derive(Clone, PartialEq, Debug)]
 pub enum Error {
@@ -9,8 +9,6 @@ pub enum Error {
     NodeUnknown(Hash256),
     InvalidFinalizedRootChange,
     InvalidNodeIndex(usize),
-    InvalidParentIndex(usize),
-    InvalidBestChildIndex(usize),
     InvalidJustifiedIndex(usize),
     InvalidBestDescendant(usize),
     InvalidParentDelta(usize),
@@ -30,7 +28,6 @@ pub enum Error {
         current_finalized_epoch: Epoch,
         new_finalized_epoch: Epoch,
     },
-    InvalidBestNode(Box<InvalidBestNodeInfo>),
     InvalidAncestorOfValidPayload {
         ancestor_block_root: Hash256,
         ancestor_payload_block_hash: ExecutionBlockHash,
@@ -74,14 +71,3 @@ impl From<ArithError> for Error {
         Error::Arith(e)
     }
 }
-
-#[derive(Clone, PartialEq, Debug)]
-pub struct InvalidBestNodeInfo {
-    pub current_slot: Slot,
-    pub start_root: Hash256,
-    pub justified_checkpoint: Checkpoint,
-    pub finalized_checkpoint: Checkpoint,
-    pub head_root: Hash256,
-    pub head_justified_checkpoint: Checkpoint,
-    pub head_finalized_checkpoint: Checkpoint,
-}

consensus/proto_array/src/proto_array.rs

@@ -1,4 +1,3 @@
-use crate::error::InvalidBestNodeInfo;
 use crate::proto_array_fork_choice::IndexedForkChoiceNode;
 use crate::{
     Block, ExecutionStatus, JustifiedBalances, LatestMessage, PayloadStatus, error::Error,
@@ -1093,28 +1092,6 @@ impl ProtoArray {
             spec,
         )?;
 
-        // Perform a sanity check that the node is indeed valid to be the head.
-        let best_node = self
-            .nodes
-            .get(best_fc_node.proto_node_index)
-            .ok_or(Error::InvalidNodeIndex(best_fc_node.proto_node_index))?;
-        if !self.node_is_viable_for_head::<E>(
-            best_node,
-            current_slot,
-            best_justified_checkpoint,
-            best_finalized_checkpoint,
-        ) {
-            return Err(Error::InvalidBestNode(Box::new(InvalidBestNodeInfo {
-                current_slot,
-                start_root: *justified_root,
-                justified_checkpoint: best_justified_checkpoint,
-                finalized_checkpoint: best_finalized_checkpoint,
-                head_root: best_node.root(),
-                head_justified_checkpoint: *best_node.justified_checkpoint(),
-                head_finalized_checkpoint: *best_node.finalized_checkpoint(),
-            })));
-        }
-
         Ok((best_fc_node.root, best_fc_node.payload_status))
     }