Web3Signer support for VC (#2522)

[EIP-3030]: https://eips.ethereum.org/EIPS/eip-3030
[Web3Signer]: https://consensys.github.io/web3signer/web3signer-eth2.html

## Issue Addressed

Resolves #2498

## Proposed Changes

Allows the VC to call out to a [Web3Signer] remote signer to obtain signatures.


## Additional Info

### Making Signing Functions `async`

To allow remote signing, I needed to make all the signing functions `async`. This caused a bit of noise where I had to convert iterators into `for` loops.

In `duties_service.rs` there was a particularly tricky case where we couldn't hold a write-lock across an `await`, so I had to first take a read-lock, then grab a write-lock.

### Move Signing from Core Executor

Whilst implementing this feature, I noticed that we signing was happening on the core tokio executor. I suspect this was causing the executor to temporarily lock and occasionally trigger some HTTP timeouts (and potentially SQL pool timeouts, but I can't verify this). Since moving all signing into blocking tokio tasks, I noticed a distinct drop in the "atttestations_http_get" metric on a Prater node:

![http_get_times](https://user-images.githubusercontent.com/6660660/132143737-82fd3836-2e7e-445b-a143-cb347783baad.png)

I think this graph indicates that freeing the core executor allows the VC to operate more smoothly.

### Refactor TaskExecutor

I noticed that the `TaskExecutor::spawn_blocking_handle` function would fail to spawn tasks if it were unable to obtain handles to some metrics (this can happen if the same metric is defined twice). It seemed that a more sensible approach would be to keep spawning tasks, but without metrics. To that end, I refactored the function so that it would still function without metrics. There are no other changes made.

## TODO

- [x] Restructure to support multiple signing methods.
- [x] Add calls to remote signer from VC.
- [x] Documentation
- [x] Test all endpoints
- [x] Test HTTPS certificate
- [x] Allow adding remote signer validators via the API
- [x] Add Altair support via [21.8.1-rc1](https://github.com/ConsenSys/web3signer/releases/tag/21.8.1-rc1)
- [x] Create issue to start using latest version of web3signer. (See #2570)

## Notes

- ~~Web3Signer doesn't yet support the Altair fork for Prater. See https://github.com/ConsenSys/web3signer/issues/423.~~
- ~~There is not yet a release of Web3Signer which supports Altair blocks. See https://github.com/ConsenSys/web3signer/issues/391.~~

common/account_utils/src/validator_definitions.rs

@@ -61,6 +61,21 @@ pub enum SigningDefinition {
         #[serde(skip_serializing_if = "Option::is_none")]
         voting_keystore_password: Option<ZeroizeString>,
     },
+    /// A validator that defers to a Web3Signer HTTP server for signing.
+    ///
+    /// https://github.com/ConsenSys/web3signer
+    #[serde(rename = "web3signer")]
+    Web3Signer {
+        url: String,
+        /// Path to a .pem file.
+        #[serde(skip_serializing_if = "Option::is_none")]
+        root_certificate_path: Option<PathBuf>,
+        /// Specifies a request timeout.
+        ///
+        /// The timeout is applied from when the request starts connecting until the response body has finished.
+        #[serde(skip_serializing_if = "Option::is_none")]
+        request_timeout_ms: Option<u64>,
+    },
 }
 
 /// A validator that may be initialized by this validator client.
@@ -116,6 +131,12 @@ impl ValidatorDefinition {
 #[derive(Default, Serialize, Deserialize)]
 pub struct ValidatorDefinitions(Vec<ValidatorDefinition>);
 
+impl From<Vec<ValidatorDefinition>> for ValidatorDefinitions {
+    fn from(vec: Vec<ValidatorDefinition>) -> Self {
+        Self(vec)
+    }
+}
+
 impl ValidatorDefinitions {
     /// Open an existing file or create a new, empty one if it does not exist.
     pub fn open_or_create<P: AsRef<Path>>(validators_dir: P) -> Result<Self, Error> {
@@ -167,11 +188,13 @@ impl ValidatorDefinitions {
         let known_paths: HashSet<&PathBuf> = self
             .0
             .iter()
-            .map(|def| match &def.signing_definition {
+            .filter_map(|def| match &def.signing_definition {
                 SigningDefinition::LocalKeystore {
                     voting_keystore_path,
                     ..
-                } => voting_keystore_path,
+                } => Some(voting_keystore_path),
+                // A Web3Signer validator does not use a local keystore file.
+                SigningDefinition::Web3Signer { .. } => None,
             })
             .collect();
 

common/eth2/src/lighthouse_vc/http_client.rs

@@ -313,6 +313,22 @@ impl ValidatorClientHttpClient {
         self.post(path, &request).await
     }
 
+    /// `POST lighthouse/validators/web3signer`
+    pub async fn post_lighthouse_validators_web3signer(
+        &self,
+        request: &[Web3SignerValidatorRequest],
+    ) -> Result<GenericResponse<ValidatorData>, Error> {
+        let mut path = self.server.full.clone();
+
+        path.path_segments_mut()
+            .map_err(|()| Error::InvalidUrl(self.server.clone()))?
+            .push("lighthouse")
+            .push("validators")
+            .push("web3signer");
+
+        self.post(path, &request).await
+    }
+
     /// `PATCH lighthouse/validators/{validator_pubkey}`
     pub async fn patch_lighthouse_validators(
         &self,

common/eth2/src/lighthouse_vc/types.rs

@@ -2,6 +2,7 @@ use account_utils::ZeroizeString;
 use eth2_keystore::Keystore;
 use graffiti::GraffitiString;
 use serde::{Deserialize, Serialize};
+use std::path::PathBuf;
 
 pub use crate::lighthouse::Health;
 pub use crate::types::{GenericResponse, VersionData};
@@ -64,3 +65,20 @@ pub struct KeystoreValidatorsPostRequest {
     pub keystore: Keystore,
     pub graffiti: Option<GraffitiString>,
 }
+
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub struct Web3SignerValidatorRequest {
+    pub enable: bool,
+    pub description: String,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub graffiti: Option<GraffitiString>,
+    pub voting_public_key: PublicKey,
+    pub url: String,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub root_certificate_path: Option<PathBuf>,
+    #[serde(default)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub request_timeout_ms: Option<u64>,
+}

common/lighthouse_metrics/src/lib.rs

@@ -283,6 +283,18 @@ pub fn set_gauge_vec(int_gauge_vec: &Result<IntGaugeVec>, name: &[&str], value:
     }
 }
 
+pub fn inc_gauge_vec(int_gauge_vec: &Result<IntGaugeVec>, name: &[&str]) {
+    if let Some(gauge) = get_int_gauge(int_gauge_vec, name) {
+        gauge.inc();
+    }
+}
+
+pub fn dec_gauge_vec(int_gauge_vec: &Result<IntGaugeVec>, name: &[&str]) {
+    if let Some(gauge) = get_int_gauge(int_gauge_vec, name) {
+        gauge.dec();
+    }
+}
+
 pub fn set_gauge(gauge: &Result<IntGauge>, value: i64) {
     if let Ok(gauge) = gauge {
         gauge.set(value);

common/task_executor/src/lib.rs

@@ -237,39 +237,33 @@ impl TaskExecutor {
     {
         let log = self.log.clone();
 
-        if let Some(metric) = metrics::get_histogram(&metrics::BLOCKING_TASKS_HISTOGRAM, &[name]) {
-            if let Some(int_gauge) = metrics::get_int_gauge(&metrics::BLOCKING_TASKS_COUNT, &[name])
-            {
-                let int_gauge_1 = int_gauge;
-                let timer = metric.start_timer();
-                let join_handle = if let Some(runtime) = self.runtime.upgrade() {
-                    runtime.spawn_blocking(task)
-                } else {
-                    debug!(self.log, "Couldn't spawn task. Runtime shutting down");
-                    return None;
-                };
+        let timer = metrics::start_timer_vec(&metrics::BLOCKING_TASKS_HISTOGRAM, &[name]);
+        metrics::inc_gauge_vec(&metrics::BLOCKING_TASKS_COUNT, &[name]);
 
-                Some(async move {
-                    let result = match join_handle.await {
-                        Ok(result) => {
-                            trace!(log, "Blocking task completed"; "task" => name);
-                            Ok(result)
-                        }
-                        Err(e) => {
-                            debug!(log, "Blocking task ended unexpectedly"; "error" => %e);
-                            Err(e)
-                        }
-                    };
-                    timer.observe_duration();
-                    int_gauge_1.dec();
-                    result
-                })
-            } else {
-                None
-            }
+        let join_handle = if let Some(runtime) = self.runtime.upgrade() {
+            runtime.spawn_blocking(task)
         } else {
-            None
-        }
+            debug!(self.log, "Couldn't spawn task. Runtime shutting down");
+            return None;
+        };
+
+        let future = async move {
+            let result = match join_handle.await {
+                Ok(result) => {
+                    trace!(log, "Blocking task completed"; "task" => name);
+                    Ok(result)
+                }
+                Err(e) => {
+                    debug!(log, "Blocking task ended unexpectedly"; "error" => %e);
+                    Err(e)
+                }
+            };
+            drop(timer);
+            metrics::dec_gauge_vec(&metrics::BLOCKING_TASKS_COUNT, &[name]);
+            result
+        };
+
+        Some(future)
     }
 
     pub fn runtime(&self) -> Weak<Runtime> {