Implement checkpoint sync (#2244)

## Issue Addressed Closes #1891 Closes #1784 ## Proposed Changes Implement checkpoint sync for Lighthouse, enabling it to start from a weak subjectivity checkpoint. ## Additional Info - [x] Return unavailable status for out-of-range blocks requested by peers (#2561) - [x] Implement sync daemon for fetching historical blocks (#2561) - [x] Verify chain hashes (either in `historical_blocks.rs` or the calling module) - [x] Consistency check for initial block + state - [x] Fetch the initial state and block from a beacon node HTTP endpoint - [x] Don't crash fetching beacon states by slot from the API - [x] Background service for state reconstruction, triggered by CLI flag or API call. Considered out of scope for this PR: - Drop the requirement to provide the `--checkpoint-block` (this would require some pretty heavy refactoring of block verification) Co-authored-by: Diva M <divma@protonmail.com>
2026-04-19 05:48:31 +00:00 · 2021-09-22 00:37:28 +00:00
parent 280e4fe23d
commit 9667dc2f03
71 changed files with 4012 additions and 459 deletions
--- a/book/src/SUMMARY.md
+++ b/book/src/SUMMARY.md
@@ -30,6 +30,7 @@
 		* [Signature Header](./api-vc-sig-header.md)
 	* [Prometheus Metrics](./advanced_metrics.md)
 * [Advanced Usage](./advanced.md)
+    * [Checkpoint Sync](./checkpoint-sync.md)
    * [Custom Data Directories](./advanced-datadir.md)
    * [Validator Graffiti](./graffiti.md)
    * [Remote Signing with Web3Signer](./validator-web3signer.md)
--- a/book/src/advanced_database.md
+++ b/book/src/advanced_database.md
@@ -25,16 +25,16 @@ some example values.

 | Use Case                | SPRP           | Yearly Disk Usage | Load Historical State |
 | ----------------------  | -------------- | ----------------- | --------------------- |
-| Block explorer/analysis | 32             | 411 GB            | 96 ms                 |
-| Default                 | 2048           | 6.4 GB            | 6 s                   |
-| Validator only          | 8192           | 1.6 GB            | 25 s                  |
+| Block explorer/analysis | 32             | 1.4 TB            | 155 ms                |
+| Default                 | 2048           | 23.1 GB           | 10.2 s                |
+| Validator only          | 8192           | 5.7 GB            | 41 s                  |

 As you can see, it's a high-stakes trade-off! The relationships to disk usage and historical state
 load time are both linear – doubling SPRP halves disk usage and doubles load time. The minimum SPRP
 is 32, and the maximum is 8192.

 The values shown in the table are approximate, calculated using a simple heuristic: each
-`BeaconState` consumes around 5MB of disk space, and each block replayed takes around 3ms.  The
+`BeaconState` consumes around 18MB of disk space, and each block replayed takes around 5ms.  The
 **Yearly Disk Usage** column shows the approx size of the freezer DB _alone_ (hot DB not included),
 and the **Load Historical State** time is the worst-case load time for a state in the last slot of
 an epoch.
--- a/book/src/api-lighthouse.md
+++ b/book/src/api-lighthouse.md
@@ -353,4 +353,58 @@ curl -X POST "http://localhost:5052/lighthouse/liveness" -d '{"indices":["0","1"
        }
    ]
 }
-```
+```
+
+### `/lighthouse/database/info`
+
+Information about the database's split point and anchor info.
+
+```bash
+curl "http://localhost:5052/lighthouse/database/info" | jq
+```
+
+```json
+{
+  "schema_version": 5,
+  "split": {
+    "slot": "2034912",
+    "state_root": "0x11c8516aa7d4d1613e84121e3a557ceca34618b4c1a38f05b66ad045ff82b33b"
+  },
+  "anchor": {
+    "anchor_slot": "2034720",
+    "oldest_block_slot": "1958881",
+    "oldest_block_parent": "0x1fd3d855d03e9df28d8a41a0f9cb9d4c540832b3ca1c3e1d7e09cd75b874cc87",
+    "state_upper_limit": "2035712",
+    "state_lower_limit": "0"
+  }
+}
+```
+
+For more information about the split point, see the [Database Configuration](./advanced_database.md)
+docs.
+
+The `anchor` will be `null` unless the node has been synced with checkpoint sync and state
+reconstruction has yet to be completed. For more information
+on the specific meanings of these fields see the docs on [Checkpoint
+Sync](./checkpoint-sync.md#reconstructing-states).
+
+### `/lighthouse/database/reconstruct`
+
+Instruct Lighthouse to begin reconstructing historic states, see
+[Reconstructing States](./checkpoint-sync.md#reconstructing-states). This is an alternative
+to the `--reconstruct-historic-states` flag.
+
+```
+curl -X POST "http://localhost:5052/lighthouse/database/reconstruct" | jq
+```
+
+```json
+"success"
+```
+
+The endpoint will return immediately. See the beacon node logs for an indication of progress.
+
+### `/lighthouse/database/historical_blocks`
+
+Manually provide `SignedBeaconBlock`s to backfill the database. This is intended
+for use by Lighthouse developers during testing only.
--- a/book/src/checkpoint-sync.md
+++ b/book/src/checkpoint-sync.md
@@ -0,0 +1,148 @@
+# Checkpoint Sync
+
+Lighthouse supports syncing from a recent finalized checkpoint. This is substantially faster
+than syncing from genesis, while still providing all the same features.
+
+If you would like to quickly get started with checkpoint sync, read the sections below on:
+
+1. [Automatic Checkpoint Sync](#automatic-checkpoint-sync)
+2. [Backfilling Blocks](#backfilling-blocks)
+
+The remaining sections are for more advanced use-cases (archival nodes).
+
+## Automatic Checkpoint Sync
+
+To begin checkpoint sync you will need HTTP API access to another synced beacon node. Enable
+checkpoint sync by providing the other beacon node's URL to `--checkpoint-sync-url`, alongside any
+other flags:
+
+```
+lighthouse bn --checkpoint-sync-url "http://remote-bn:5052" ...
+```
+
+Lighthouse will print a message to indicate that checkpoint sync is being used:
+
+```
+INFO Starting checkpoint sync                remote_url: http://remote-bn:8000/, service: beacon
+```
+
+After a short time (usually less than a minute), it will log the details of the checkpoint
+loaded from the remote beacon node:
+
+```
+INFO Loaded checkpoint block and state       state_root: 0xe8252c68784a8d5cc7e5429b0e95747032dd1dcee0d1dc9bdaf6380bf90bc8a6, block_root: 0x5508a20147299b1a7fe9dbea1a8b3bf979f74c52e7242039bd77cbff62c0695a, slot: 2034720, service: beacon
+```
+
+> **Security Note**: You should cross-reference the `block_root` and `slot` of the loaded checkpoint
+> against a trusted source like a friend's node, or a block explorer.
+
+Once the checkpoint is loaded Lighthouse will sync forwards to the head of the chain.
+
+If a validator client is connected to the node then it will be able to start completing its duties
+as soon as forwards sync completes.
+
+## Backfilling Blocks
+
+Once forwards sync completes, Lighthouse will commence a "backfill sync" to download the blocks
+from the checkpoint back to genesis.
+
+The beacon node will log messages similar to the following each minute while it completes backfill
+sync:
+
+```
+INFO Downloading historical blocks  est_time: 5 hrs 0 mins, speed: 111.96 slots/sec, distance: 2020451 slots (40 weeks 0 days), service: slot_notifier
+```
+
+Once backfill is complete, a `INFO Historical block download complete` log will be emitted.
+
+## FAQ
+
+1. What if I have an existing database? How can I use checkpoint sync?
+
+The existing beacon database needs to be deleted before Lighthouse will attempt checkpoint sync.
+You can do this by providing the `--purge-db` flag, or by manually deleting `<DATADIR>/beacon`.
+
+2. Why is checkpoint sync faster?
+
+Checkpoint sync prioritises syncing to the head of the chain quickly so that the node can perform
+its duties. Additionally, it only has to perform lightweight verification of historic blocks:
+it checks the hash chain integrity & proposer signature rather than computing the full state
+transition.
+
+3. Is checkpoint sync less secure?
+
+No, in fact it is more secure! Checkpoint sync guards against long-range attacks that
+genesis sync does not. This is due to a property of Proof of Stake consensus known as [Weak
+Subjectivity][weak-subj].
+
+## Reconstructing States
+
+> This section is only relevant if you are interested in running an archival node for analysis
+> purposes.
+
+After completing backfill sync the node's database will differ from a genesis-synced node in the
+lack of historic states. _You do not need these states to run a staking node_, but they are required
+for historical API calls (as used by block explorers and researchers).
+
+You can opt-in to reconstructing all of the historic states by providing the
+`--reconstruct-historic-states` flag to the beacon node at any point (before, during or after sync).
+
+The database keeps track of three markers to determine the availability of historic blocks and
+states:
+
+* `oldest_block_slot`: All blocks with slots less than or equal to this value are available in the
+  database. Additionally, the genesis block is always available.
+* `state_lower_limit`: All states with slots _less than or equal to_ this value are available in
+  the database. The minimum value is 0, indicating that the genesis state is always available.
+* `state_upper_limit`: All states with slots _greater than or equal to_ this value are available
+  in the database.
+
+Reconstruction runs from the state lower limit to the upper limit, narrowing the window of
+unavailable states as it goes. It will log messages like the following to show its progress:
+
+```
+INFO State reconstruction in progress        remaining: 747519, slot: 466944, service: freezer_db
+```
+
+Important information to be aware of:
+
+* Reconstructed states will consume several gigabytes or hundreds of gigabytes of disk space,
+  depending on the [database configuration used](./advanced_database.md).
+* Reconstruction will only begin once backfill sync has completed and `oldest_block_slot` is
+  equal to 0.
+* While reconstruction is running the node will temporarily pause migrating new data to the
+  freezer database. This will lead to the database increasing in size temporarily (by a few GB per
+  day) until state reconstruction completes.
+* It is safe to interrupt state reconstruction by gracefully terminating the node – it will pick up
+  from where it left off when it restarts.
+* You can start reconstruction from the HTTP API, and view its progress. See the
+  [`/lighthouse/database`](./api-lighthouse.md) APIs.
+
+For more information on historic state storage see the
+[Database Configuration](./advanced_database.md) page.
+
+## Manual Checkpoint Sync
+
+> This section is only relevant if you want to manually provide the checkpoint state and
+> block instead of fetching them from a URL.
+
+To manually specify a checkpoint use the following two flags:
+
+* `--checkpoint-state`: accepts an SSZ-encoded `BeaconState` blob
+* `--checkpoint-block`: accepts an SSZ-encoded `SignedBeaconBlock` blob
+
+_Both_ the state and block must be provided and **must** adhere to the [Alignment
+Requirements](#alignment-requirements) described below.
+
+### Alignment Requirements
+
+* The block must be a finalized block from an epoch boundary, i.e. `block.slot() % 32 == 0`.
+* The state must be the state corresponding to `block` with `state.slot() == block.slot()`
+  and `state.hash_tree_root() == block.state_root()`.
+
+These requirements are imposed to align with Lighthouse's database schema, and notably exclude
+finalized blocks from skipped slots. You can avoid alignment issues by using
+[Automatic Checkpoint Sync](#automatic-checkpoint-sync), which will search for a suitable block
+and state pair.
+
+[weak-subj]: https://blog.ethereum.org/2014/11/25/proof-stake-learned-love-weak-subjectivity/