Web3Signer support for VC (#2522)

[EIP-3030]: https://eips.ethereum.org/EIPS/eip-3030
[Web3Signer]: https://consensys.github.io/web3signer/web3signer-eth2.html

## Issue Addressed

Resolves #2498

## Proposed Changes

Allows the VC to call out to a [Web3Signer] remote signer to obtain signatures.


## Additional Info

### Making Signing Functions `async`

To allow remote signing, I needed to make all the signing functions `async`. This caused a bit of noise where I had to convert iterators into `for` loops.

In `duties_service.rs` there was a particularly tricky case where we couldn't hold a write-lock across an `await`, so I had to first take a read-lock, then grab a write-lock.

### Move Signing from Core Executor

Whilst implementing this feature, I noticed that we signing was happening on the core tokio executor. I suspect this was causing the executor to temporarily lock and occasionally trigger some HTTP timeouts (and potentially SQL pool timeouts, but I can't verify this). Since moving all signing into blocking tokio tasks, I noticed a distinct drop in the "atttestations_http_get" metric on a Prater node:

![http_get_times](https://user-images.githubusercontent.com/6660660/132143737-82fd3836-2e7e-445b-a143-cb347783baad.png)

I think this graph indicates that freeing the core executor allows the VC to operate more smoothly.

### Refactor TaskExecutor

I noticed that the `TaskExecutor::spawn_blocking_handle` function would fail to spawn tasks if it were unable to obtain handles to some metrics (this can happen if the same metric is defined twice). It seemed that a more sensible approach would be to keep spawning tasks, but without metrics. To that end, I refactored the function so that it would still function without metrics. There are no other changes made.

## TODO

- [x] Restructure to support multiple signing methods.
- [x] Add calls to remote signer from VC.
- [x] Documentation
- [x] Test all endpoints
- [x] Test HTTPS certificate
- [x] Allow adding remote signer validators via the API
- [x] Add Altair support via [21.8.1-rc1](https://github.com/ConsenSys/web3signer/releases/tag/21.8.1-rc1)
- [x] Create issue to start using latest version of web3signer. (See #2570)

## Notes

- ~~Web3Signer doesn't yet support the Altair fork for Prater. See https://github.com/ConsenSys/web3signer/issues/423.~~
- ~~There is not yet a release of Web3Signer which supports Altair blocks. See https://github.com/ConsenSys/web3signer/issues/391.~~

book/src/SUMMARY.md

@@ -32,6 +32,7 @@
 * [Advanced Usage](./advanced.md)
     * [Custom Data Directories](./advanced-datadir.md)
     * [Validator Graffiti](./graffiti.md)
+    * [Remote Signing with Web3Signer](./validator-web3signer.md)
     * [Database Configuration](./advanced_database.md)
     * [Advanced Networking](./advanced_networking.md)
     * [Running a Slasher](./slasher.md)

book/src/api-vc-endpoints.md

@@ -434,3 +434,43 @@ Typical Responses | 200
     ]
 }
 ```
+
+## `POST /lighthouse/validators/web3signer`
+
+Create any number of new validators, all of which will refer to a
+[Web3Signer](https://docs.web3signer.consensys.net/en/latest/) server for signing.
+
+### HTTP Specification
+
+| Property | Specification |
+| --- |--- |
+Path | `/lighthouse/validators/web3signer`
+Method | POST
+Required Headers | [`Authorization`](./api-vc-auth-header.md)
+Typical Responses | 200, 400
+
+### Example Request Body
+
+```json
+[
+    {
+        "enable": true,
+        "description": "validator_one",
+        "graffiti": "Mr F was here",
+        "voting_public_key": "0xa062f95fee747144d5e511940624bc6546509eeaeae9383257a9c43e7ddc58c17c2bab4ae62053122184c381b90db380",
+        "url": "http://path-to-web3signer.com",
+        "root_certificate_path": "/path/on/vc/filesystem/to/certificate.pem",
+        "request_timeout_ms": 12000
+    }
+]
+```
+
+The following fields may be omitted or nullified to obtain default values:
+
+- `graffiti`
+- `root_certificate_path`
+- `request_timeout_ms`
+
+### Example Response Body
+
+*No data is included in the response body.*

book/src/validator-web3signer.md

@@ -0,0 +1,56 @@
+# Remote Signing with Web3Signer
+
+[Web3Signer]: https://docs.web3signer.consensys.net/en/latest/
+[Consensys]: https://github.com/ConsenSys/
+[Teku]: https://github.com/consensys/teku
+
+[Web3Signer] is a tool by Consensys which allows *remote signing*. Remote signing is when a
+Validator Client (VC) out-sources the signing of messages to remote server (e.g., via HTTPS). This
+means that the VC does not hold the validator private keys.
+
+## Warnings
+
+Using a remote signer comes with risks, please read the following two warnings before proceeding:
+
+### Remote signing is complex and risky
+
+Remote signing is generally only desirable for enterprise users or users with unique security
+requirements. Most users will find the separation between the Beacon Node (BN) and VC to be
+sufficient *without* introducing a remote signer.
+
+**Using a remote signer introduces a new set of security and slashing risks and should only be
+undertaken by advanced users who fully understand the risks.**
+
+### Web3Signer is not maintained by Lighthouse
+
+The [Web3Signer] tool is maintained by [Consensys], the same team that maintains [Teku]. The
+Lighthouse team (Sigma Prime) does not maintain Web3Signer or make any guarantees about its safety
+or effectiveness.
+
+## Usage
+
+A remote signing validator is added to Lighthouse in much the same way as one that uses a local
+keystore, via the [`validator_definitions.yml`](./validator-management.md) file or via the `POST
+/lighthouse/validators/web3signer` API endpoint.
+
+Here is an example of a `validator_definitions.yml` file containing one validator which uses a
+remote signer:
+
+```yaml
+---
+- enabled: true
+  voting_public_key: "0xa5566f9ec3c6e1fdf362634ebec9ef7aceb0e460e5079714808388e5d48f4ae1e12897fed1bea951c17fa389d511e477"
+  type: web3signer
+  url: "https://my-remote-signer.com:1234"
+  root_certificate_path: /home/paul/my-certificates/my-remote-signer.pem
+```
+
+When using this file, the Lighthouse VC will perform duties for the `0xa5566..` validator and defer
+to the `https://my-remote-signer.com:1234` server to obtain any signatures. It will load a
+"self-signed" SSL certificate from `/home/paul/my-certificates/my-remote-signer.pem` (on the
+filesystem of the VC) to encrypt the communications between the VC and Web3Signer.
+
+> The `request_timeout_ms` key can also be specified. Use this key to override the default timeout
+> with a new timeout in milliseconds. This is the timeout before requests to Web3Signer are
+> considered to be failures. Setting a value that it too-long may create contention and late duties
+> in the VC.  Setting it too short will result in failed signatures and therefore missed duties.