Web3Signer support for VC (#2522)

[EIP-3030]: https://eips.ethereum.org/EIPS/eip-3030
[Web3Signer]: https://consensys.github.io/web3signer/web3signer-eth2.html

## Issue Addressed

Resolves #2498

## Proposed Changes

Allows the VC to call out to a [Web3Signer] remote signer to obtain signatures.


## Additional Info

### Making Signing Functions `async`

To allow remote signing, I needed to make all the signing functions `async`. This caused a bit of noise where I had to convert iterators into `for` loops.

In `duties_service.rs` there was a particularly tricky case where we couldn't hold a write-lock across an `await`, so I had to first take a read-lock, then grab a write-lock.

### Move Signing from Core Executor

Whilst implementing this feature, I noticed that we signing was happening on the core tokio executor. I suspect this was causing the executor to temporarily lock and occasionally trigger some HTTP timeouts (and potentially SQL pool timeouts, but I can't verify this). Since moving all signing into blocking tokio tasks, I noticed a distinct drop in the "atttestations_http_get" metric on a Prater node:

![http_get_times](https://user-images.githubusercontent.com/6660660/132143737-82fd3836-2e7e-445b-a143-cb347783baad.png)

I think this graph indicates that freeing the core executor allows the VC to operate more smoothly.

### Refactor TaskExecutor

I noticed that the `TaskExecutor::spawn_blocking_handle` function would fail to spawn tasks if it were unable to obtain handles to some metrics (this can happen if the same metric is defined twice). It seemed that a more sensible approach would be to keep spawning tasks, but without metrics. To that end, I refactored the function so that it would still function without metrics. There are no other changes made.

## TODO

- [x] Restructure to support multiple signing methods.
- [x] Add calls to remote signer from VC.
- [x] Documentation
- [x] Test all endpoints
- [x] Test HTTPS certificate
- [x] Allow adding remote signer validators via the API
- [x] Add Altair support via [21.8.1-rc1](https://github.com/ConsenSys/web3signer/releases/tag/21.8.1-rc1)
- [x] Create issue to start using latest version of web3signer. (See #2570)

## Notes

- ~~Web3Signer doesn't yet support the Altair fork for Prater. See https://github.com/ConsenSys/web3signer/issues/423.~~
- ~~There is not yet a release of Web3Signer which supports Altair blocks. See https://github.com/ConsenSys/web3signer/issues/391.~~

validator_client/src/http_api/create_validator.rs

@@ -1,4 +1,5 @@
 use crate::ValidatorStore;
+use account_utils::validator_definitions::{SigningDefinition, ValidatorDefinition};
 use account_utils::{
     eth2_wallet::{bip39::Mnemonic, WalletBuilder},
     random_mnemonic, random_password, ZeroizeString,
@@ -21,7 +22,7 @@ use validator_dir::Builder as ValidatorDirBuilder;
 ///
 /// If `key_derivation_path_offset` is supplied then the EIP-2334 validator index will start at
 /// this point.
-pub async fn create_validators<P: AsRef<Path>, T: 'static + SlotClock, E: EthSpec>(
+pub async fn create_validators_mnemonic<P: AsRef<Path>, T: 'static + SlotClock, E: EthSpec>(
     mnemonic_opt: Option<Mnemonic>,
     key_derivation_path_offset: Option<u32>,
     validator_requests: &[api_types::ValidatorRequest],
@@ -159,3 +160,33 @@ pub async fn create_validators<P: AsRef<Path>, T: 'static + SlotClock, E: EthSpe
 
     Ok((validators, mnemonic))
 }
+
+pub async fn create_validators_web3signer<T: 'static + SlotClock, E: EthSpec>(
+    validator_requests: &[api_types::Web3SignerValidatorRequest],
+    validator_store: &ValidatorStore<T, E>,
+) -> Result<(), warp::Rejection> {
+    for request in validator_requests {
+        let validator_definition = ValidatorDefinition {
+            enabled: request.enable,
+            voting_public_key: request.voting_public_key.clone(),
+            graffiti: request.graffiti.clone(),
+            description: request.description.clone(),
+            signing_definition: SigningDefinition::Web3Signer {
+                url: request.url.clone(),
+                root_certificate_path: request.root_certificate_path.clone(),
+                request_timeout_ms: request.request_timeout_ms,
+            },
+        };
+        validator_store
+            .add_validator(validator_definition)
+            .await
+            .map_err(|e| {
+                warp_utils::reject::custom_server_error(format!(
+                    "failed to initialize validator: {:?}",
+                    e
+                ))
+            })?;
+    }
+
+    Ok(())
+}

validator_client/src/http_api/mod.rs

@@ -4,7 +4,7 @@ mod tests;
 
 use crate::ValidatorStore;
 use account_utils::mnemonic_from_phrase;
-use create_validator::create_validators;
+use create_validator::{create_validators_mnemonic, create_validators_web3signer};
 use eth2::lighthouse_vc::types::{self as api_types, PublicKey, PublicKeyBytes};
 use lighthouse_version::version_with_platform;
 use serde::{Deserialize, Serialize};
@@ -273,14 +273,15 @@ pub fn serve<T: 'static + SlotClock + Clone, E: EthSpec>(
              runtime: Weak<Runtime>| {
                 blocking_signed_json_task(signer, move || {
                     if let Some(runtime) = runtime.upgrade() {
-                        let (validators, mnemonic) = runtime.block_on(create_validators(
-                            None,
-                            None,
-                            &body,
-                            &validator_dir,
-                            &validator_store,
-                            &spec,
-                        ))?;
+                        let (validators, mnemonic) =
+                            runtime.block_on(create_validators_mnemonic(
+                                None,
+                                None,
+                                &body,
+                                &validator_dir,
+                                &validator_store,
+                                &spec,
+                            ))?;
                         let response = api_types::PostValidatorsResponseData {
                             mnemonic: mnemonic.into_phrase().into(),
                             validators,
@@ -322,14 +323,15 @@ pub fn serve<T: 'static + SlotClock + Clone, E: EthSpec>(
                                     e
                                 ))
                             })?;
-                        let (validators, _mnemonic) = runtime.block_on(create_validators(
-                            Some(mnemonic),
-                            Some(body.key_derivation_path_offset),
-                            &body.validators,
-                            &validator_dir,
-                            &validator_store,
-                            &spec,
-                        ))?;
+                        let (validators, _mnemonic) =
+                            runtime.block_on(create_validators_mnemonic(
+                                Some(mnemonic),
+                                Some(body.key_derivation_path_offset),
+                                &body.validators,
+                                &validator_dir,
+                                &validator_store,
+                                &spec,
+                            ))?;
                         Ok(api_types::GenericResponse::from(validators))
                     } else {
                         Err(warp_utils::reject::custom_server_error(
@@ -416,6 +418,33 @@ pub fn serve<T: 'static + SlotClock + Clone, E: EthSpec>(
             },
         );
 
+    // POST lighthouse/validators/web3signer
+    let post_validators_web3signer = warp::path("lighthouse")
+        .and(warp::path("validators"))
+        .and(warp::path("web3signer"))
+        .and(warp::path::end())
+        .and(warp::body::json())
+        .and(validator_store_filter.clone())
+        .and(signer.clone())
+        .and(runtime_filter.clone())
+        .and_then(
+            |body: Vec<api_types::Web3SignerValidatorRequest>,
+             validator_store: Arc<ValidatorStore<T, E>>,
+             signer,
+             runtime: Weak<Runtime>| {
+                blocking_signed_json_task(signer, move || {
+                    if let Some(runtime) = runtime.upgrade() {
+                        runtime.block_on(create_validators_web3signer(&body, &validator_store))?;
+                        Ok(())
+                    } else {
+                        Err(warp_utils::reject::custom_server_error(
+                            "Runtime shutdown".into(),
+                        ))
+                    }
+                })
+            },
+        );
+
     // PATCH lighthouse/validators/{validator_pubkey}
     let patch_validators = warp::path("lighthouse")
         .and(warp::path("validators"))
@@ -484,7 +513,8 @@ pub fn serve<T: 'static + SlotClock + Clone, E: EthSpec>(
                 .or(warp::post().and(
                     post_validators
                         .or(post_validators_keystore)
-                        .or(post_validators_mnemonic),
+                        .or(post_validators_mnemonic)
+                        .or(post_validators_web3signer),
                 ))
                 .or(warp::patch().and(patch_validators)),
         )

validator_client/src/http_api/tests.rs

@@ -4,7 +4,8 @@
 use crate::doppelganger_service::DoppelgangerService;
 use crate::{
     http_api::{ApiSecret, Config as HttpConfig, Context},
-    Config, InitializedValidators, ValidatorDefinitions, ValidatorStore,
+    initialized_validators::InitializedValidators,
+    Config, ValidatorDefinitions, ValidatorStore,
 };
 use account_utils::{
     eth2_wallet::WalletBuilder, mnemonic_from_phrase, random_mnemonic, random_password,
@@ -27,6 +28,7 @@ use std::marker::PhantomData;
 use std::net::Ipv4Addr;
 use std::sync::Arc;
 use std::time::Duration;
+use task_executor::TaskExecutor;
 use tempfile::{tempdir, TempDir};
 use tokio::runtime::Runtime;
 use tokio::sync::oneshot;
@@ -41,6 +43,7 @@ struct ApiTester {
     url: SensitiveUrl,
     _server_shutdown: oneshot::Sender<()>,
     _validator_dir: TempDir,
+    _runtime_shutdown: exit_future::Signal,
 }
 
 // Builds a runtime to be used in the testing configuration.
@@ -85,6 +88,10 @@ impl ApiTester {
         let slot_clock =
             TestingSlotClock::new(Slot::new(0), Duration::from_secs(0), Duration::from_secs(1));
 
+        let (runtime_shutdown, exit) = exit_future::signal();
+        let (shutdown_tx, _) = futures::channel::mpsc::channel(1);
+        let executor = TaskExecutor::new(runtime.clone(), exit, log.clone(), shutdown_tx);
+
         let validator_store = ValidatorStore::<_, E>::new(
             initialized_validators,
             slashing_protection,
@@ -92,6 +99,7 @@ impl ApiTester {
             spec,
             Some(Arc::new(DoppelgangerService::new(log.clone()))),
             slot_clock,
+            executor,
             log.clone(),
         );
 
@@ -141,6 +149,7 @@ impl ApiTester {
             client,
             url,
             _server_shutdown: shutdown_tx,
+            _runtime_shutdown: runtime_shutdown,
         }
     }
 
@@ -425,6 +434,40 @@ impl ApiTester {
         self
     }
 
+    pub async fn create_web3signer_validators(self, s: Web3SignerValidatorScenario) -> Self {
+        let initial_vals = self.vals_total();
+        let initial_enabled_vals = self.vals_enabled();
+
+        let request: Vec<_> = (0..s.count)
+            .map(|i| {
+                let kp = Keypair::random();
+                Web3SignerValidatorRequest {
+                    enable: s.enabled,
+                    description: format!("{}", i),
+                    graffiti: None,
+                    voting_public_key: kp.pk,
+                    url: format!("http://signer_{}.com/", i),
+                    root_certificate_path: None,
+                    request_timeout_ms: None,
+                }
+            })
+            .collect();
+
+        self.client
+            .post_lighthouse_validators_web3signer(&request)
+            .await
+            .unwrap_err();
+
+        assert_eq!(self.vals_total(), initial_vals + s.count);
+        if s.enabled {
+            assert_eq!(self.vals_enabled(), initial_enabled_vals + s.count);
+        } else {
+            assert_eq!(self.vals_enabled(), initial_enabled_vals);
+        };
+
+        self
+    }
+
     pub async fn set_validator_enabled(self, index: usize, enabled: bool) -> Self {
         let validator = &self.client.get_lighthouse_validators().await.unwrap().data[index];
 
@@ -480,6 +523,11 @@ struct KeystoreValidatorScenario {
     correct_password: bool,
 }
 
+struct Web3SignerValidatorScenario {
+    count: usize,
+    enabled: bool,
+}
+
 #[test]
 fn invalid_pubkey() {
     let runtime = build_runtime();
@@ -677,3 +725,22 @@ fn keystore_validator_creation() {
             .assert_validators_count(2);
     });
 }
+
+#[test]
+fn web3signer_validator_creation() {
+    let runtime = build_runtime();
+    let weak_runtime = Arc::downgrade(&runtime);
+    runtime.block_on(async {
+        ApiTester::new(weak_runtime)
+            .await
+            .assert_enabled_validators_count(0)
+            .assert_validators_count(0)
+            .create_web3signer_validators(Web3SignerValidatorScenario {
+                count: 1,
+                enabled: true,
+            })
+            .await
+            .assert_enabled_validators_count(1)
+            .assert_validators_count(1);
+    });
+}