* persist light client updates
* update beacon chain to serve light client updates
* resolve todos
* cache best update
* extend cache parts
* is better light client update
* resolve merge conflict
* initial api changes
* add lc update db column
* fmt
* added tests
* add sim
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* fix some weird issues with the simulator
* tests
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* test changes
* merge conflict
* testing
* started work on ef tests and some code clean up
* update tests
* linting
* noop pre altair, were still failing on electra though
* allow for zeroed light client header
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* merge unstable
* remove unwraps
* remove unwraps
* fetch bootstrap without always querying for state
* storing bootstrap parts in db
* mroe code cleanup
* test
* prune sync committee branches from dropped chains
* Update light_client_update.rs
* merge unstable
* move functionality to helper methods
* refactor is best update fn
* refactor is best update fn
* improve organization of light client server cache logic
* fork diget calc, and only spawn as many blcoks as we need for the lc update test
* resovle merge conflict
* add electra bootstrap logic, add logic to cache current sync committee
* add latest sync committe branch cache
* fetch lc update from the cache if it exists
* fmt
* Fix beacon_chain tests
* Add debug code to update ranking_order ef test
* Fix compare code
* merge conflicts
* merge conflict
* add better error messaging
* resolve merge conflicts
* remove lc update from basicsim
* rename sync comittte variable and fix persist condition
* refactor get_light_client_update logic
* add better comments, return helpful error messages over http and rpc
* pruning canonical non checkpoint slots
* fix test
* rerun test
* update pruning logic, add tests
* fix tests
* fix imports
* fmt
* refactor db code
* Refactor db method
* Refactor db method
* add additional comments
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-bootstrap
* fix merge
* linting
* merge conflict
* prevent overflow
* enable lc server for http api tests
* fix tests
* remove prints
* remove warning
* revert change
* Store changes to persist data columns.
Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com>
* Update to use `eip7594_fork_epoch` for data column slot in Store.
* Fix formatting.
* Merge branch 'unstable' into data-columns-store
# Conflicts:
# beacon_node/store/src/lib.rs
# consensus/types/src/chain_spec.rs
* Minor refactor.
* Merge branch 'unstable' into data-columns-store
# Conflicts:
# beacon_node/store/src/metrics.rs
* Init data colum info at PeerDAS epoch instead of Deneb fork epoch. Address review comments.
* Remove Deneb-related comments
## Issue Addressed
Closes#4817.
## Proposed Changes
- Fill in the linear block roots array between 0 and the slot of the first block (e.g. slots 0 and 1 on Holesky).
- Backport the `--freezer`, `--skip` and `--limit` options for `lighthouse db inspect` from tree-states. This allows us to easily view the database corruption of 4817 using `lighthouse db inspect --network holesky --freezer --column bbr --output values --limit 2`.
- Backport the `iter_column_from` change and `MemoryStore` overhaul from tree-states. These are required to enable `lighthouse db inspect`.
- Rework `freezer_upper_limit` to allow state lookups for slots below the `state_lower_limit`. Currently state lookups will fail until state reconstruction completes entirely.
There is a new regression test for the main bug, but no test for the `freezer_upper_limit` fix because we don't currently support running state reconstruction partially (see #3026). This will be fixed once we merge `tree-states`! In lieu of an automated test, I've tested manually on a Holesky node while it was reconstructing.
## Additional Info
Users who backfilled Holesky to slot 0 (e.g. using `--reconstruct-historic-states`) need to either:
- Re-sync from genesis.
- Re-sync using checkpoint sync and the changes from this PR.
Due to the recency of the Holesky genesis, writing a custom pass to fix up broken databases (which would require its own thorough testing) was deemed unnecessary. This is the primary reason for this PR being marked `backwards-incompat`.
This will create few conflicts with Deneb, which I've already resolved on `tree-states-deneb` and will be happy to backport to Deneb once this PR is merged to unstable.
* remove protoc and token from network tests github action
* delete unused beacon chain methods
* downgrade writing blobs to store log
* reduce diff in block import logic
* remove some todo's and deneb built in network
* remove unnecessary error, actually use some added metrics
* remove some metrics, fix missing components on publish funcitonality
* fix status tests
* rename sidecar by root to blobs by root
* clean up some metrics
* remove unnecessary feature gate from attestation subnet tests, clean up blobs by range response code
* pawan's suggestion in `protocol_info`, peer score in matching up batch sync block and blobs
* fix range tests for deneb
* pub block and blob db cache behind the same mutex
* remove unused errs and an empty file
* move sidecar trait to new file
* move types from payload to eth2 crate
* update comment and add flag value name
* make function private again, remove allow unused
* use reth rlp for tx decoding
* fix compile after merge
* rename kzg commitments
* cargo fmt
* remove unused dep
* Update beacon_node/execution_layer/src/lib.rs
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
* Update beacon_node/beacon_processor/src/lib.rs
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
* pawan's suggestiong for vec capacity
* cargo fmt
* Revert "use reth rlp for tx decoding"
This reverts commit 5181837d81.
* remove reth rlp
---------
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
## Proposed Changes
Reduce post-merge disk usage by not storing finalized execution payloads in Lighthouse's database.
⚠️ **This is achieved in a backwards-incompatible way for networks that have already merged** ⚠️. Kiln users and shadow fork enjoyers will be unable to downgrade after running the code from this PR. The upgrade migration may take several minutes to run, and can't be aborted after it begins.
The main changes are:
- New column in the database called `ExecPayload`, keyed by beacon block root.
- The `BeaconBlock` column now stores blinded blocks only.
- Lots of places that previously used full blocks now use blinded blocks, e.g. analytics APIs, block replay in the DB, etc.
- On finalization:
- `prune_abanonded_forks` deletes non-canonical payloads whilst deleting non-canonical blocks.
- `migrate_db` deletes finalized canonical payloads whilst deleting finalized states.
- Conversions between blinded and full blocks are implemented in a compositional way, duplicating some work from Sean's PR #3134.
- The execution layer has a new `get_payload_by_block_hash` method that reconstructs a payload using the EE's `eth_getBlockByHash` call.
- I've tested manually that it works on Kiln, using Geth and Nethermind.
- This isn't necessarily the most efficient method, and new engine APIs are being discussed to improve this: https://github.com/ethereum/execution-apis/pull/146.
- We're depending on the `ethers` master branch, due to lots of recent changes. We're also using a workaround for https://github.com/gakonst/ethers-rs/issues/1134.
- Payload reconstruction is used in the HTTP API via `BeaconChain::get_block`, which is now `async`. Due to the `async` fn, the `blocking_json` wrapper has been removed.
- Payload reconstruction is used in network RPC to serve blocks-by-{root,range} responses. Here the `async` adjustment is messier, although I think I've managed to come up with a reasonable compromise: the handlers take the `SendOnDrop` by value so that they can drop it on _task completion_ (after the `fn` returns). Still, this is introducing disk reads onto core executor threads, which may have a negative performance impact (thoughts appreciated).
## Additional Info
- [x] For performance it would be great to remove the cloning of full blocks when converting them to blinded blocks to write to disk. I'm going to experiment with a `put_block` API that takes the block by value, breaks it into a blinded block and a payload, stores the blinded block, and then re-assembles the full block for the caller.
- [x] We should measure the latency of blocks-by-root and blocks-by-range responses.
- [x] We should add integration tests that stress the payload reconstruction (basic tests done, issue for more extensive tests: https://github.com/sigp/lighthouse/issues/3159)
- [x] We should (manually) test the schema v9 migration from several prior versions, particularly as blocks have changed on disk and some migrations rely on being able to load blocks.
Co-authored-by: Paul Hauner <paul@paulhauner.com>
## Issue Addressed
Successor to #2431
## Proposed Changes
* Add a `BlockReplayer` struct to abstract over the intricacies of calling `per_slot_processing` and `per_block_processing` while avoiding unnecessary tree hashing.
* Add a variant of the forwards state root iterator that does not require an `end_state`.
* Use the `BlockReplayer` when reconstructing states in the database. Use the efficient forwards iterator for frozen states.
* Refactor the iterators to remove `Arc<HotColdDB>` (this seems to be neater than making _everything_ an `Arc<HotColdDB>` as I did in #2431).
Supplying the state roots allow us to avoid building a tree hash cache at all when reconstructing historic states, which saves around 1 second flat (regardless of `slots-per-restore-point`). This is a small percentage of worst-case state load times with 200K validators and SPRP=2048 (~15s vs ~16s) but a significant speed-up for more frequent restore points: state loads with SPRP=32 should be now consistently <500ms instead of 1.5s (a ~3x speedup).
## Additional Info
Required by https://github.com/sigp/lighthouse/pull/2628
## Issue Addressed
Closes#1891Closes#1784
## Proposed Changes
Implement checkpoint sync for Lighthouse, enabling it to start from a weak subjectivity checkpoint.
## Additional Info
- [x] Return unavailable status for out-of-range blocks requested by peers (#2561)
- [x] Implement sync daemon for fetching historical blocks (#2561)
- [x] Verify chain hashes (either in `historical_blocks.rs` or the calling module)
- [x] Consistency check for initial block + state
- [x] Fetch the initial state and block from a beacon node HTTP endpoint
- [x] Don't crash fetching beacon states by slot from the API
- [x] Background service for state reconstruction, triggered by CLI flag or API call.
Considered out of scope for this PR:
- Drop the requirement to provide the `--checkpoint-block` (this would require some pretty heavy refactoring of block verification)
Co-authored-by: Diva M <divma@protonmail.com>
## Issue Addressed
Closes#1787
## Proposed Changes
* Abstract the `ValidatorPubkeyCache` over a "backing" which is either a file (legacy), or the database.
* Implement a migration from schema v2 to schema v3, whereby the contents of the cache file are copied to the DB, and then the file is deleted. The next release to include this change must be a minor version bump, and we will need to warn users of the inability to downgrade (this is our first DB schema change since mainnet genesis).
* Move the schema migration code from the `store` crate into the `beacon_chain` crate so that it can access the datadir and the `ValidatorPubkeyCache`, etc. It gets injected back into the `store` via a closure (similar to what we do in fork choice).
## Issue Addressed
Closes#673
## Proposed Changes
Store a schema version in the database so that future releases can check they're running against a compatible database version. This would also enable automatic migration on breaking database changes, but that's left as future work.
The database config is also stored in the database so that the `slots_per_restore_point` value can be checked for consistency, which closes#673
* Improve error handling in block iteration
* Introduce atomic DB operations
* Fix race condition
An invariant was violated: For every block hash in head_tracker, that
block is accessible from the store.
* Store dht enrs on shutdown
* Load enrs on startup and add tests
* Remove enr_entries from behavior
* Move all dht persisting logic to `NetworkService`
* Move `PersistedDht` from eth2-libp2p to network crate
* Add test to confirm dht persistence
* Add logging
* Remove extra call to beacon_chain persist
* Expose only mutable `add_enr` method from behaviour
* Fix tests
* Fix merge errors
* Sparse hot DB and block root tree
* Fix store_tests
* Ensure loads of hot states on boundaries are fast
* Milder error for unaligned finalized blocks
* Implement freezer database for state vectors
* Improve BeaconState safe accessors
And fix a bug in the compact committees accessor.
* Banish dodgy type bounds back to gRPC
* Clean up
* Switch to exclusive end points in chunked vec
* Cleaning up and start of tests
* Randao fix, more tests
* Fix unsightly hack
* Resolve test FIXMEs
* Config file support
* More clean-ups, migrator beginnings
* Finish migrator, integrate into BeaconChain
* Fixups
* Fix store tests
* Fix BeaconChain tests
* Fix LMD GHOST tests
* Address review comments, delete 'static bounds
* Cargo format
* Address review comments
* Fix LMD ghost tests
* Update to spec v0.9.0
* Update to v0.9.1
* Bump spec tags for v0.9.1
* Formatting, fix CI failures
* Resolve accidental KeyPair merge conflict
* Document new BeaconState functions
* Fix incorrect cache drops in `advance_caches`
* Update fork choice for v0.9.1
* Clean up some FIXMEs
* Fix a few docs/logs
* Update for new builder paradigm, spec changes
* Freezer DB integration into BeaconNode
* Cleaning up
* This works, clean it up
* Cleanups
* Fix and improve store tests
* Refine store test
* Delete unused beacon_chain_builder.rs
* Fix CLI
* Store state at split slot in hot database
* Make fork choice lookup fast again
* Store freezer DB split slot in the database
* Handle potential div by 0 in chunked_vector
* Exclude committee caches from freezer DB
* Remove FIXME about long-running test
