Rework Validator Client fallback mechanism (#4393)

* Rework Validator Client fallback mechanism

* Add CI workflow for fallback simulator

* Tie-break with sync distance for non-synced nodes

* Fix simulator

* Cleanup unused code

* More improvements

* Add IsOptimistic enum for readability

* Use configurable sync distance tiers

* Fix tests

* Combine status and health and improve logging

* Fix nodes not being marked as available

* Fix simulator

* Fix tests again

* Increase fallback simulator tolerance

* Add http api endpoint

* Fix todos and tests

* Update simulator

* Merge branch 'unstable' into vc-fallback

* Add suggestions

* Add id to ui endpoint

* Remove unnecessary clones

* Formatting

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Fix flag tests

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Fix conflicts

* Merge branch 'unstable' into vc-fallback

* Remove unnecessary pubs

* Simplify `compute_distance_tier` and reduce notifier awaits

* Use the more descriptive `user_index` instead of `id`

* Combine sync distance tolerance flags into one

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* wip

* Use new simulator from unstable

* Fix cli text

* Remove leftover files

* Remove old commented code

* Merge branch 'unstable' into vc-fallback

* Update cli text

* Silence candidate errors when pre-genesis

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Retry on failure

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Remove disable_run_on_all

* Remove unused error variant

* Fix out of date comment

* Merge branch 'unstable' into vc-fallback

* Remove unnecessary as_u64

* Remove more out of date comments

* Use tokio RwLock and remove parking_lot

* Merge branch 'unstable' into vc-fallback

* Formatting

* Ensure nodes are still added to total when not available

* Allow VC to detect when BN comes online

* Fix ui endpoint

* Don't have block_service as an Option

* Merge branch 'unstable' into vc-fallback

* Clean up lifetimes and futures

* Revert "Don't have block_service as an Option"

This reverts commit b5445a09e9.

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Improve rwlock sanitation using clones

* Merge branch 'unstable' into vc-fallback

* Drop read lock immediately by cloning the vec.

validator_client/src/cli.rs

@@ -444,6 +444,33 @@ pub fn cli_app() -> Command {
                 .help_heading(FLAG_HEADER)
                 .display_order(0)
         )
+        .arg(
+            Arg::new("beacon-nodes-sync-tolerances")
+                .long("beacon-nodes-sync-tolerances")
+                .value_name("SYNC_TOLERANCES")
+                .help("A comma-separated list of 3 values which sets the size of each sync distance range when \
+                    determining the health of each connected beacon node. \
+                    The first value determines the `Synced` range. \
+                    If a connected beacon node is synced to within this number of slots it is considered 'Synced'. \
+                    The second value determines the `Small` sync distance range. \
+                    This range starts immediately after the `Synced` range. \
+                    The third value determines the `Medium` sync distance range. \
+                    This range starts immediately after the `Small` range. \
+                    Any sync distance value beyond that is considered `Large`. \
+                    For example, a value of `8,8,48` would have ranges like the following: \
+                    `Synced`: 0..=8 \
+                    `Small`: 9..=16 \
+                    `Medium`: 17..=64 \
+                    `Large`: 65.. \
+                    These values are used to determine what ordering beacon node fallbacks are used in. \
+                    Generally, `Synced` nodes are preferred over `Small` and so on. \
+                    Nodes in the `Synced` range will tie-break based on their ordering in `--beacon-nodes`. \
+                    This ensures the primary beacon node is prioritised. \
+                    [default: 8,8,48]")
+                .action(ArgAction::Set)
+                .help_heading(FLAG_HEADER)
+                .display_order(0)
+        )
         .arg(
             Arg::new("disable-slashing-protection-web3signer")
                 .long("disable-slashing-protection-web3signer")