Add documentation for VC API /lighthouse/beacon/health (#6653)

Changes the endpoint to get fallback health information from `/lighthouse/ui/fallback_health` to `/lighthouse/beacon/health`. This more accurately describes that the endpoint is related to the connected beacon nodes and also matched the `/lighthouse/beacon/update` endpoint being added in #6551.

Adds documentation for both fallback health and the endpoint to the Lighthouse book.

book/src/advanced_redundancy.md

@@ -39,9 +39,6 @@ There are a few interesting properties about the list of `--beacon-nodes`:
  earlier in the list.
 - *Synced is preferred*: the validator client prefers a synced beacon node over
  one that is still syncing.
-- *Failure is sticky*: if a beacon node fails, it will be flagged as offline
-    and won't be retried again for the rest of the slot (12 seconds). This helps prevent the impact
-    of time-outs and other lengthy errors.
 
 > Note: When supplying multiple beacon nodes the `http://localhost:5052` address must be explicitly
 > provided (if it is desired). It will only be used as default if no `--beacon-nodes` flag is
@@ -76,6 +73,22 @@ Prior to v3.2.0 fallback beacon nodes also required the `--subscribe-all-subnets
 now broadcast subscriptions to all connected beacon nodes by default. This broadcast behaviour
 can be disabled using the `--broadcast none` flag for `lighthouse vc`.
 
+### Fallback Health
+
+Since v6.0.0, the validator client will be more aggressive in switching to a fallback node. To do this,
+it uses the concept of "Health". Every slot, the validator client checks each connected beacon node
+to determine which node is the "Healthiest". In general, the validator client will prefer nodes
+which are synced, have synced execution layers and which are not currently optimistically
+syncing.
+
+Sync distance is separated out into 4 tiers: "Synced", "Small", "Medium", "Large". Nodes are then
+sorted into tiers based onto sync distance and execution layer status. You can use the
+`--beacon-nodes-sync-tolerances` to change how many slots wide each tier is. In the case where
+multiple nodes fall into the same tier, user order is used to tie-break.
+
+To see health information for each connected node, you can use the
+[`/lighthouse/beacon/health` API endpoint](./api_vc_endpoints.md#get-lighthousebeaconhealth).
+
 ### Broadcast modes
 
 Since v4.6.0, the Lighthouse VC can be configured to broadcast messages to all configured beacon

book/src/api_vc_endpoints.md

@@ -18,6 +18,7 @@
 | [`POST /lighthouse/validators/mnemonic`](#post-lighthousevalidatorsmnemonic) | Create a new validator from an existing mnemonic. |
 | [`POST /lighthouse/validators/web3signer`](#post-lighthousevalidatorsweb3signer) | Add web3signer validators. |
 | [`GET /lighthouse/logs`](#get-lighthouselogs) | Get logs |
+| [`GET /lighthouse/beacon/health`](#get-lighthousebeaconhealth) | Get health information for each connected beacon node. |
 
 The query to Lighthouse API endpoints requires authorization, see [Authorization Header](./api_vc_auth_header.md).
 
@@ -816,3 +817,56 @@ logs emitted are INFO level or higher.
   }
 }
 ```
+
+## `GET /lighthouse/beacon/health`
+
+Provides information about the sync status and execution layer health of each connected beacon node.
+For more information about how to interpret the beacon node health, see [Fallback Health](./advanced_redundancy.md#fallback-health).
+
+### HTTP Specification
+
+| Property          | Specification                              |
+|-------------------|--------------------------------------------|
+| Path              | `/lighthouse/beacon/health`                |
+| Method            | GET                                        |
+| Required Headers  | [`Authorization`](./api_vc_auth_header.md) |
+| Typical Responses | 200, 400                                   |
+
+Command:
+
+```bash
+DATADIR=/var/lib/lighthouse
+curl -X GET http://localhost:5062/lighthouse/beacon/health \
+ -H "Authorization: Bearer $(cat ${DATADIR}/validators/api-token.txt)" | jq
+ ```
+
+### Example Response Body
+
+```json
+{
+    "data": {
+        "beacon_nodes": [
+            {
+                "index": 0,
+                "endpoint": "http://localhost:5052",
+                "health": {
+                    "user_index": 0,
+                    "head": 10500000,
+                    "optimistic_status": "No",
+                    "execution_status": "Healthy",
+                    "health_tier": {
+                        "tier": 1,
+                        "sync_distance": 0,
+                        "distance_tier": "Synced"
+                    }
+                }
+            },
+            {
+                "index": 1,
+                "endpoint": "http://fallbacks-r.us",
+                "health": "Offline"
+            }
+        ]
+    }
+}
+```

validator_client/http_api/src/lib.rs

@@ -418,10 +418,10 @@ pub fn serve<T: 'static + SlotClock + Clone, E: EthSpec>(
             },
         );
 
-    // GET lighthouse/ui/fallback_health
-    let get_lighthouse_ui_fallback_health = warp::path("lighthouse")
-        .and(warp::path("ui"))
-        .and(warp::path("fallback_health"))
+    // GET lighthouse/beacon/health
+    let get_lighthouse_beacon_health = warp::path("lighthouse")
+        .and(warp::path("beacon"))
+        .and(warp::path("health"))
         .and(warp::path::end())
         .and(block_service_filter.clone())
         .then(|block_filter: BlockService<T, E>| async move {
@@ -1294,7 +1294,7 @@ pub fn serve<T: 'static + SlotClock + Clone, E: EthSpec>(
                         .or(get_lighthouse_validators_pubkey)
                         .or(get_lighthouse_ui_health)
                         .or(get_lighthouse_ui_graffiti)
-                        .or(get_lighthouse_ui_fallback_health)
+                        .or(get_lighthouse_beacon_health)
                         .or(get_fee_recipient)
                         .or(get_gas_limit)
                         .or(get_graffiti)