lighthouse/book/src/validator_manager_move.md

# Moving Validators

The `lighthouse validator-manager move` command uses the VC HTTP API to move
validators from one VC (the "src" VC) to another VC (the "dest" VC). The move
operation is *comprehensive*; it will:

- Disable the validators on the src VC.
- Remove the validator keystores from the src VC file system.
- Export the slashing database records for the appropriate validators from the src VC to the dest VC.
- Enable the validators on the dest VC.
- Generally result in very little or no validator downtime.

It is capable of moving all validators on the src VC, a count of validators or
a list of pubkeys.

The `move` command is only guaranteed to work between two Lighthouse VCs (i.e.,
there is no guarantee that the commands will work between Lighthouse and Teku, for instance).

The `move` command only supports moving validators using a keystore on the local
file system, it does not support `Web3Signer` validators.

Although all efforts are taken to avoid it, it's possible for the `move` command
to fail in a way that removes the validator from the src VC without adding it to the
dest VC. Therefore, it is recommended to **never use the `move` command without
having a backup of all validator keystores (e.g. the mnemonic).**

## Simple Example

The following command will move all validators from the VC running at
`http://localhost:6062` to the VC running at `http://localhost:5062`.

```bash
lighthouse \
	validator-manager \
	move \
	--src-vc-url http://localhost:6062 \
	--src-vc-token ~/src-token.txt \
	--dest-vc-url http://localhost:5062 \
	--dest-vc-token ~/.lighthouse/mainnet/validators/api-token.txt \
	--validators all
```

## Detailed Guide

This guide describes the steps to move validators between two validator clients (VCs) which are
able to SSH between each other. This guide assumes experience with the Linux command line and SSH
connections.

There will be two VCs in this example:

- The *source* VC which contains the validators/keystores to be moved.
- The *destination* VC which is to take the validators/keystores from the source.

There will be two hosts in this example:

- Host 1 (*"source host"*): Is running the `src-vc`.
- Host 2 (*"destination host"*): Is running the `dest-vc`.

The example assumes
that Host 1 is able to SSH to Host 2.

In reality, many host configurations are possible. For example:

- Both VCs on the same host.
- Both VCs on different hosts and the `validator-manager` being used on a third host.

### 1. Configure the Source VC

The source VC needs to have the following flags at a minimum:

- `--http`
- `--http-allow-keystore-export`

Therefore, the source VC command might look like:

```bash
lighthouse \
    vc \
    --http \
    --http-allow-keystore-export
```

### 2. Configure the Destination VC

The destination VC needs to have the following flags at a minimum:

- `--http`
- `--enable-doppelganger-protection`

Therefore, the destination VC command might look like:

```bash
lighthouse \
    vc \
    --http \
    --enable-doppelganger-protection
```

> The `--enable-doppelganger-protection` flag is not *strictly* required, however
> it is recommended for an additional layer of safety. It will result in 2-3
> epochs of downtime for the validator after it is moved, which is generally an
> inconsequential cost in lost rewards or penalties.
>
> Optionally, users can add the `--http-store-passwords-in-secrets-dir` flag if they'd like to have
> the import validator keystore passwords stored in separate files rather than in the
> `validator-definitions.yml` file. If you don't know what this means, you can safely omit the flag.

### 3. Obtain the Source API Token

The VC API is protected by an *API token*. This is stored in a file on each of the hosts. Since
we'll be running our command on the destination host, it will need to have the API token for the
source host on its file-system.

On the **source host**, find the location of the `api-token.txt` file and copy the contents. The
location of the file varies, but it is located in the "validator directory" of your data directory,
alongside validator keystores. For example: `~/.lighthouse/mainnet/validators/api-token.txt`. If you are unsure of the `api-token.txt` path, you can run `curl http://localhost:5062/lighthouse/auth` which will show the path.

Copy the contents of that file into a new file on the **destination host** at `~/src-token.txt`. The
API token is a random string, e.g., `hGut6B8uEujufDXSmZsT0thnxvdvKFBvh`.

### 4. Create an SSH Tunnel

In the **source host**, open a terminal window, SSH to the **destination host** and establish a reverse-SSH connection
between the **destination host** and the **source host**.

```bash
ssh dest-host
ssh -L 6062:localhost:5062 src-host
```

It's important that you leave this session open throughout the rest of this tutorial. If you close
this terminal window then the connection between the destination and source host will be lost.

### 5. Move

With the SSH tunnel established between the `dest-host` and `src-host`, from the **destination
host** run the command to move the validators:

```bash
lighthouse \
	validator-manager \
	move \
	--src-vc-url http://localhost:6062 \
	--src-vc-token ~/src-token.txt \
	--dest-vc-url http://localhost:5062 \
	--dest-vc-token ~/.lighthouse/mainnet/validators/api-token.txt \
	--validators all
```

The command will provide information about the progress of the operation and
emit `Done.` when the operation has completed successfully. For example:

```bash
Running validator manager for mainnet network
Validator client is reachable at http://localhost:5062/ and reports 2 validators
Validator client is reachable at http://localhost:6062/ and reports 0 validators
Moved keystore 1 of 2
Moved keystore 2 of 2
Done.
```

At the same time, `lighthouse vc` will log:

```bash
INFO Importing keystores via standard HTTP API, count: 1
INFO Enabled validator                       voting_pubkey: 0xab6e29f1b98fedfca878edce2b471f1b5ee58ee4c3bd216201f98254ef6f6eac40a53d74c8b7da54f51d3e85cacae92f, signing_method: local_keystore
INFO Modified key_cache saved successfully
```

Once the operation completes successfully, there is nothing else to be done. The
validators have been removed from the `src-host` and enabled at the `dest-host`.
If the `--enable-doppelganger-protection` flag was used it may take 2-3 epochs
for the validators to start attesting and producing blocks on the `dest-host`.
If you would only like to move some validators, you can replace the flag `--validators all` with one or more validator public keys. For example:

```bash
lighthouse \
	validator-manager \
	move \
	--src-vc-url http://localhost:6062 \
	--src-vc-token ~/src-token.txt \
	--dest-vc-url http://localhost:5062 \
	--dest-vc-token ~/.lighthouse/mainnet/validators/api-token.txt \
	--validators 0x9096aab771e44da149bd7c9926d6f7bb96ef465c0eeb4918be5178cd23a1deb4aec232c61d85ff329b54ed4a3bdfff3a,0x90fc4f72d898a8f01ab71242e36f4545aaf87e3887be81632bb8ba4b2ae8fb70753a62f866344d7905e9a07f5a9cdda1
```

> Note: If you have the `validator-monitor-auto` turned on, the source beacon node may still be reporting the attestation status of the validators that have been moved:

```
INFO Previous epoch attestation(s) success   validators: ["validator_index"], epoch: 100000, service: val_mon, service: beacon
```

> This is fine as the validator monitor does not know that the validators have been moved (it *does not* mean that the validators have attested twice for the same slot). A restart of the beacon node will resolve this.

Any errors encountered during the operation should include information on how to
proceed. Assistance is also available on our
[Discord](https://discord.gg/cyAszAh).