Commit Graph

806 Commits

Author SHA1 Message Date
Michael Sproul
bde0f1ef0b Merge remote-tracking branch 'origin/release-v7.0.0' into unstable 2025-03-29 13:01:58 +11:00
Pawan Dhananjay
54aef2d043 Admin add/remove peer (#7198)
N/A


  Adds endpoints to add and remove trusted peers from the http api. The added peers are trusted peers so they won't be disconnected for bad scores. We try to maintain a connection to the peer in case they disconnect from us by trying to dial it every heartbeat.
2025-03-28 12:59:09 +00:00
Michael Sproul
6d5a2be7f9 Release v7.0.0-beta.5 (#7210)
New release for Pectra-enabled networks.
2025-03-27 03:42:34 +00:00
Eitan Seri-Levi
cbf1c04a14 resolve merge conflicts between untstable and release-v7.0.0 2025-03-23 11:09:02 -06:00
Michael Sproul
04868027a6 Release v7.0.0-beta.4 (#7162)
New release for Hoodi testnet including clean versions of fixes from `holesky-rescue`.
2025-03-20 05:12:19 +00:00
Eitan Seri-Levi
27aabe8159 Pseudo finalization endpoint (#7103)
This is a backport of:
-  https://github.com/sigp/lighthouse/pull/7059
- https://github.com/sigp/lighthouse/pull/7071

For:
- https://github.com/sigp/lighthouse/issues/7039


  Introduce a new lighthouse endpoint that allows a user to force a pseudo finalization. This migrates data to the freezer db and prunes sidechains which may help reduce disk space issues on non finalized networks like Holesky

We also ban peers that send us blocks that conflict with the manually finalized checkpoint.

There were some CI fixes in https://github.com/sigp/lighthouse/pull/7071 that I tried including here

Co-authored with: @jimmygchen  @pawanjay176 @michaelsproul
2025-03-18 05:21:05 +00:00
Michael Sproul
58482586f5 Support Hoodi testnet (#7145)
Hardcode config for the upcoming `hoodi` testnet so we can run with `--network hoodi`.
2025-03-18 02:10:24 +00:00
Jun Song
50b5a72c58 feat: implement new beacon APIs(accessors for pending_deposits/pending_partial_withdrawals) (#7006)
Resolves #7003


  Added two endpoints as https://github.com/ethereum/beacon-APIs/pull/500 proposed:
- `/eth/v1/beacon/states/{state_id}/pending_deposits`
- `/eth/v1/beacon/states/{state_id}/pending_partial_withdrawals`
2025-03-17 01:46:50 +00:00
ThreeHrSleep
5cda1641ea Log file appender initialization errors properly (#7137)
It'll help us debug the issue with rolling file appender initialization (`Failed to initialize libp2p rolling file appender`)

cc: @macladson
2025-03-17 01:25:55 +00:00
Daniel Knopik
574b204bdb decouple eth2 from store and lighthouse_network (#6680)
- #6452 (partially)


  Remove dependencies on `store` and `lighthouse_network`  from `eth2`. This was achieved as follows:

- depend on `enr` and `multiaddr` directly instead of using `lighthouse_network`'s reexports.
- make `lighthouse_network` responsible for converting between API and internal types.
- in two cases, remove complex internal types and use the generic `serde_json::Value` instead - this is not ideal, but should be fine for now, as this affects two internal non-spec endpoints which are meant for debugging, unstable, and subject to change without notice anyway. Inspired by #6679. The alternative is to move all relevant types to `eth2` or `types` instead - what do you think?
2025-03-14 16:44:48 +00:00
ThreeHrSleep
d60c24ef1c Integrate tracing (#6339)
Tracing Integration
- [reference](5bbf1859e9/projects/project-ideas.md (L297))


  - [x] replace slog & log with tracing throughout the codebase
- [x] implement custom crit log
- [x] make relevant changes in the formatter
- [x] replace sloggers
- [x] re-write SSE logging components

cc: @macladson @eserilev
2025-03-12 22:31:05 +00:00
Paul Hauner
8d1abce26e Bump SSZ version for larger bitfield SmallVec (#6915)
NA


  Bumps the `ethereum_ssz` version, along with other crates that share the dep.

Primarily, this give us bitfields which can store 128 bytes on the stack before allocating, rather than 32 bytes (https://github.com/sigp/ethereum_ssz/pull/38). The validator count has increase massively since we set it at 32 bytes, so aggregation bitfields (et al) now require a heap allocation. This new value of 128 should get us to ~2m active validators.
2025-03-10 08:18:33 +00:00
Eitan Seri-Levi
066f28770f Schedule Chiado testnet Electra hard fork (#7074) 2025-03-05 06:21:13 +00:00
Pawan Dhananjay
b3b6aea1c5 Rust 1.85 lints (#7019)
N/A


  2 changes:
1. Replace Option::map_or(true, ...) with is_none_or(...)
2. Remove unnecessary `Into::into` blocks where the type conversion is apparent from the types
2025-02-24 02:36:13 +00:00
Michael Sproul
1888be554c Release v7.0.0-beta.0 (#6962)
New release for Electra on Holesky and Sepolia.

Includes PRs:

- https://github.com/sigp/lighthouse/pull/6808
- https://github.com/sigp/lighthouse/pull/6914
- https://github.com/sigp/lighthouse/pull/6949
- https://github.com/sigp/lighthouse/pull/6950
- https://github.com/sigp/lighthouse/pull/6958
2025-02-13 03:06:20 +00:00
Eitan Seri-Levi
e3e21f7516 Schedule Sepolia and Holesky Electra forks (#6949) 2025-02-10 01:27:00 +00:00
Eitan Seri-Levi
1e2b547b35 Add builder SSZ flow (#6859) 2025-02-03 06:07:42 +00:00
Eitan Seri-Levi
276eda3dfe POST /eth/v2/beacon/pool/attestations bugfixes (#6867) 2025-01-31 00:20:44 +00:00
Jimmy Chen
70194dfc6a Implement PeerDAS Fulu fork activation (#6795)
Addresses #6706


  This PR activates PeerDAS at the Fulu fork epoch instead of `EIP_7594_FORK_EPOCH`. This means we no longer support testing PeerDAS with Deneb / Electrs, as it's now part of a hard fork.
2025-01-30 07:01:34 +00:00
Janick Martinez Esturo
d297d08c6b Increase jemalloc aarch64 page size limit (#5244) (#6831)
#5244


  Pass `JEMALLOC_SYS_WITH_LG_PAGE=16` env to aarch64 cross-compilation to support systems with up to 64-KiB page sizes. This is backwards-compatible for the current (most usual) 4-KiB systems.
2025-01-30 05:14:57 +00:00
Jimmy Chen
2b6ec96b4c Add MetaData V3 support to node/identity API (#6827)
* Add metadata v3 support to `node/identity` api.
2025-01-22 04:05:29 +00:00
JKinc
33c1648022 Add EIP-7636 support (#6793)
* Add eip7636 support

* Add `version()` to the `lighthouse_version` crate and make the `enr.rs` file use it.

* Hardcode version, Add `client_name()`, remove unneeded flag.

* Make it use the new function.

* Make cargo fmt zip it
2025-01-21 00:24:14 +00:00
Eitan Seri-Levi
06329ec2d1 SingleAttestation implementation (#6488)
* First pass

* Add restrictions to RuntimeVariableList api

* Use empty_uninitialized and fix warnings

* Fix some todos

* Merge branch 'unstable' into max-blobs-preset

* Fix take impl on RuntimeFixedList

* cleanup

* Fix test compilations

* Fix some more tests

* Fix test from unstable

* Merge branch 'unstable' into max-blobs-preset

* SingleAttestation

* Add post attestation v2 endpoint logic to attestation service

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation

* Implement "Bugfix and more withdrawal tests"

* Implement "Add missed exit checks to consolidation processing"

* Implement "Update initial earliest_exit_epoch calculation"

* Implement "Limit consolidating balance by validator.effective_balance"

* Implement "Use 16-bit random value in validator filter"

* Implement "Do not change creds type on consolidation"

* some tests and fixed attestqtion calc

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation

* Rename PendingPartialWithdraw index field to validator_index

* Skip slots to get test to pass and add TODO

* Implement "Synchronously check all transactions to have non-zero length"

* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset

* Remove footgun function

* Minor simplifications

* Move from preset to config

* Fix typo

* Revert "Remove footgun function"

This reverts commit de01f923c7.

* Try fixing tests

* Implement "bump minimal preset MAX_BLOB_COMMITMENTS_PER_BLOCK and KZG_COMMITMENT_INCLUSION_PROOF_DEPTH"

* Thread through ChainSpec

* Fix release tests

* Move RuntimeFixedVector into module and rename

* Add test

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation

* Added more test coverage, simplified Attestation conversion, and other minor refactors

* Removed unusued codepaths

* Fix failing test

* Implement "Remove post-altair `initialize_beacon_state_from_eth1` from specs"

* Update preset YAML

* Remove empty RuntimeVarList awefullness

* Make max_blobs_per_block a config parameter (#6329)

Squashed commit of the following:

commit 04b3743ec1
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 17:36:58 2025 +1100

    Add test

commit 440e854199
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 17:24:50 2025 +1100

    Move RuntimeFixedVector into module and rename

commit f66e179a40
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 17:17:17 2025 +1100

    Fix release tests

commit e4bfe71cd1
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 17:05:30 2025 +1100

    Thread through ChainSpec

commit 063b79c16a
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 15:32:16 2025 +1100

    Try fixing tests

commit 88bedf09bc
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 15:04:37 2025 +1100

    Revert "Remove footgun function"

    This reverts commit de01f923c7.

commit 32483d385b
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 15:04:32 2025 +1100

    Fix typo

commit 2e86585b47
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 15:04:15 2025 +1100

    Move from preset to config

commit 1095d60a40
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 14:38:40 2025 +1100

    Minor simplifications

commit de01f923c7
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 14:06:57 2025 +1100

    Remove footgun function

commit 0c2c8c4224
Merge: 21ecb58ff f51a292f7
Author: Michael Sproul <michael@sigmaprime.io>
Date:   Mon Jan 6 14:02:50 2025 +1100

    Merge remote-tracking branch 'origin/unstable' into max-blobs-preset

commit f51a292f77
Author: Daniel Knopik <107140945+dknopik@users.noreply.github.com>
Date:   Fri Jan 3 20:27:21 2025 +0100

    fully lint only explicitly to avoid unnecessary rebuilds (#6753)

    * fully lint only explicitly to avoid unnecessary rebuilds

commit 7e0cddef32
Author: Akihito Nakano <sora.akatsuki@gmail.com>
Date:   Tue Dec 24 10:38:56 2024 +0900

    Make sure we have fanout peers when publish (#6738)

    * Ensure that `fanout_peers` is always non-empty if it's `Some`

commit 21ecb58ff8
Merge: 2fcb2935e 9aefb5539
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Mon Oct 21 14:46:00 2024 -0700

    Merge branch 'unstable' into max-blobs-preset

commit 2fcb2935ec
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Fri Sep 6 18:28:31 2024 -0700

    Fix test from unstable

commit 12c6ef118a
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Wed Sep 4 16:16:36 2024 -0700

    Fix some more tests

commit d37733b846
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Wed Sep 4 12:47:36 2024 -0700

    Fix test compilations

commit 52bb581e07
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Tue Sep 3 18:38:19 2024 -0700

    cleanup

commit e71020e3e6
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Tue Sep 3 17:16:10 2024 -0700

    Fix take impl on RuntimeFixedList

commit 13f9bba647
Merge: 60100fc6b 4e675cf5d
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Tue Sep 3 16:08:59 2024 -0700

    Merge branch 'unstable' into max-blobs-preset

commit 60100fc6be
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Fri Aug 30 16:04:11 2024 -0700

    Fix some todos

commit a9cb329a22
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Fri Aug 30 15:54:00 2024 -0700

    Use empty_uninitialized and fix warnings

commit 4dc6e6515e
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Fri Aug 30 15:53:18 2024 -0700

    Add restrictions to RuntimeVariableList api

commit 25feedfde3
Author: Pawan Dhananjay <pawandhananjay@gmail.com>
Date:   Thu Aug 29 16:11:19 2024 -0700

    First pass

* Fix tests

* Implement max_blobs_per_block_electra

* Fix config issues

* Simplify BlobSidecarListFromRoot

* Disable PeerDAS tests

* Cleanup single attestation imports

* Fix some single attestation network plumbing

* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset

* Bump quota to account for new target (6)

* Remove clone

* Fix issue from review

* Try to remove ugliness

* Merge branch 'unstable' into max-blobs-preset

* Merge remote-tracking branch 'origin/unstable' into electra-alpha10

* Merge commit '04b3743ec1e0b650269dd8e58b540c02430d1c0d' into electra-alpha10

* Merge remote-tracking branch 'pawan/max-blobs-preset' into electra-alpha10

* Update tests to v1.5.0-beta.0

* Merge remote-tracking branch 'origin/electra-alpha10' into single_attestation

* Fix some tests

* Cargo fmt

* lint

* fmt

* Resolve merge conflicts

* Merge branch 'electra-alpha10' of https://github.com/sigp/lighthouse into single_attestation

* lint

* Linting

* fmt

* Merge branch 'electra-alpha10' of https://github.com/sigp/lighthouse into single_attestation

* Fmt

* Fix test and add TODO

* Gracefully handle slashed proposers in fork choice tests

* Merge remote-tracking branch 'origin/unstable' into electra-alpha10

* Keep latest changes from max_blobs_per_block PR in codec.rs

* Revert a few more regressions and add a comment

* Merge branch 'electra-alpha10' of https://github.com/sigp/lighthouse into single_attestation

* Disable more DAS tests

* Improve validator monitor test a little

* Make test more robust

* Fix sync test that didn't understand blobs

* Fill out cropped comment

* Merge remote-tracking branch 'origin/electra-alpha10' into single_attestation

* Merge remote-tracking branch 'origin/unstable' into single_attestation

* Merge remote-tracking branch 'origin/unstable' into single_attestation

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation

* publish_attestations should accept Either<Attestation,SingleAttestation>

* log an error when failing to convert to SingleAttestation

* Use Cow to avoid clone

* Avoid reconverting to SingleAttestation

* Tweak VC error message

* update comments

* update comments

* pass in single attestation as ref to subnetid calculation method

* Improved API, new error variants and other minor tweaks

* Fix single_attestation event topic boilerplate

* fix sse event failure

* Add single_attestation event topic test coverage
2025-01-16 18:27:08 +00:00
Daniel Knopik
669932aa67 Misc. dependency cleanup (#6810)
* remove ensure_dir_exists (2 deps saved)

* group UNHANDLED_ERRORs into a generic (2 deps saved)

* Introduce separate `health_metrics` crate

* separate health_metrics crate

* remove metrics from warp_utils

* move ProcessHealth::observe and SystemHealth::observe to health_metrics

* fix errors

* nitpick `Cargo.toml`s

---------

Co-authored-by: Daniel Knopik <daniel@dknopik.de>
# Conflicts:
#	Cargo.toml
2025-01-16 01:48:50 +00:00
Jimmy Chen
e98209d118 Implement PeerDAS subnet decoupling (aka custody groups) (#6736)
* Implement PeerDAS subnet decoupling (aka custody groups).

* Merge branch 'unstable' into decouple-subnets

* Refactor feature testing for spec tests (#6737)

Squashed commit of the following:

commit 898d05ee17
Merge: ffbd25e2b 7e0cddef3
Author: Jimmy Chen <jchen.tc@gmail.com>
Date:   Tue Dec 24 14:41:19 2024 +1100

    Merge branch 'unstable' into refactor-ef-tests-features

commit ffbd25e2be
Author: Jimmy Chen <jchen.tc@gmail.com>
Date:   Tue Dec 24 14:40:38 2024 +1100

    Fix `SszStatic` tests for PeerDAS: exclude eip7594 test vectors when testing Electra types.

commit aa593cf35c
Author: Jimmy Chen <jchen.tc@gmail.com>
Date:   Fri Dec 20 12:08:54 2024 +1100

    Refactor spec testing for features and simplify usage.

* Fix build.

* Add input validation and improve arithmetic handling when calculating custody groups.

* Address review comments re code style consistency.

* Merge branch 'unstable' into decouple-subnets

# Conflicts:
#	beacon_node/beacon_chain/src/kzg_utils.rs
#	beacon_node/beacon_chain/src/observed_data_sidecars.rs
#	beacon_node/lighthouse_network/src/discovery/subnet_predicate.rs
#	common/eth2_network_config/built_in_network_configs/chiado/config.yaml
#	common/eth2_network_config/built_in_network_configs/gnosis/config.yaml
#	common/eth2_network_config/built_in_network_configs/holesky/config.yaml
#	common/eth2_network_config/built_in_network_configs/mainnet/config.yaml
#	common/eth2_network_config/built_in_network_configs/sepolia/config.yaml
#	consensus/types/src/chain_spec.rs

* Update consensus/types/src/chain_spec.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into decouple-subnets

* Update error handling.

* Address review comment.

* Merge remote-tracking branch 'origin/unstable' into decouple-subnets

# Conflicts:
#	consensus/types/src/chain_spec.rs

* Update PeerDAS spec tests to `1.5.0-beta.0` and fix failing unit tests.

* Merge remote-tracking branch 'origin/unstable' into decouple-subnets

# Conflicts:
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs
2025-01-15 07:40:26 +00:00
Mac L
348fbdb838 Add missing crates to cargo workspace (#6774)
* Add the remaining crates to cargo workspace

* Merge branch 'unstable' into add-remaining-crates-workspace
2025-01-10 07:35:05 +00:00
Pawan Dhananjay
05727290fb Make max_blobs_per_block a config parameter (#6329)
* First pass

* Add restrictions to RuntimeVariableList api

* Use empty_uninitialized and fix warnings

* Fix some todos

* Merge branch 'unstable' into max-blobs-preset

* Fix take impl on RuntimeFixedList

* cleanup

* Fix test compilations

* Fix some more tests

* Fix test from unstable

* Merge branch 'unstable' into max-blobs-preset

* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset

* Remove footgun function

* Minor simplifications

* Move from preset to config

* Fix typo

* Revert "Remove footgun function"

This reverts commit de01f923c7.

* Try fixing tests

* Thread through ChainSpec

* Fix release tests

* Move RuntimeFixedVector into module and rename

* Add test

* Remove empty RuntimeVarList awefullness

* Fix tests

* Simplify BlobSidecarListFromRoot

* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset

* Bump quota to account for new target (6)

* Remove clone

* Fix issue from review

* Try to remove ugliness

* Merge branch 'unstable' into max-blobs-preset

* Fix max value

* Fix doctest

* Fix formatting

* Fix max check

* Delete hardcoded max_blobs_per_block in RPC limits

* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
2025-01-10 06:34:58 +00:00
Mac L
ecdf2d891f Add Fulu boilerplate (#6695)
* Add Fulu boilerplate

* Add more boilerplate

* Change fulu_time to osaka_time

* Merge branch 'unstable' into fulu-boilerplate

* Fix tests

* Merge branch 'unstable' into fulu-boilerplate

* More test fixes

* Apply suggestions

* Remove `get_payload` boilerplate

* Add lightclient fulu types and fix beacon-chain-tests

* Disable Fulu in ef-tests

* Reduce boilerplate for future forks

* Small fixes

* One more fix

* Apply suggestions

* Merge branch 'unstable' into fulu-boilerplate

* Fix lints
2025-01-10 05:25:23 +00:00
Pawan Dhananjay
1f6850fae2 Rust 1.84 lints (#6781)
* Fix few lints

* Fix remaining lints

* Use fully qualified syntax
2025-01-10 01:13:29 +00:00
Mac L
b2b1faad4e Enforce alphabetically ordered cargo deps (#6678)
* Enforce alphabetically ordered cargo deps

* Fix test-suite

* Another CI fix

* Merge branch 'unstable' into cargo-sort

* Fix conflicts

* Merge remote-tracking branch 'origin/unstable' into cargo-sort
2024-12-19 05:46:03 +00:00
Pawan Dhananjay
2662dc7f8f Fix Sse client api (#6685)
* Use reqwest eventsource for get_events api

* await for Event::Open before returning stream

* fmt

* Merge branch 'unstable' into sse-client-fix

* Ignore lint
2024-12-18 00:05:58 +00:00
Daniel Knopik
02cb2d68ff Enable lints for tests only running optimized (#6664)
* enable linting optimized-only tests

* fix automatically fixable or obvious lints

* fix suspicious_open_options by removing manual options

* fix `await_holding_lock`s

* avoid failing lint due to now disabled `#[cfg(debug_assertions)]`

* reduce future sizes in tests

* fix accidently flipped assert logic

* restore holding lock for web3signer download

* Merge branch 'unstable' into lint-opt-tests
2024-12-17 00:40:35 +00:00
Michael Sproul
a6de0d5e12 Merge remote-tracking branch 'origin/stable' into back-merge-6.0.1 2024-12-16 15:37:52 +11:00
Michael Sproul
0d90135047 Release v6.0.1 (#6659)
* Release v6.0.1
2024-12-16 03:03:22 +00:00
Age Manning
c3a0757ad2 Correct /nat API for libp2p (#6677)
* Fix nat API
2024-12-15 23:16:53 +00:00
Daniel Knopik
a2b00090fd Remove ZeroizeString in favour of Zeroizing<String> (#6661)
* Remove ZeroizeString in favour of Zeroizing<String>

* cargo fmt

* remove unrelated line that slipped in

* Update beacon_node/store/Cargo.toml

thanks michael!

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Merge branch 'unstable' into remove-zeroizedstring
2024-12-11 23:51:20 +00:00
Michael Sproul
c042dc14d7 Release v6.0.0 (#6605)
* Release v6.0.0
2024-12-02 04:00:51 +00:00
Michael Sproul
fa6c4c02a3 Fix Rust 1.83 Clippy lints (#6629)
* Fix Rust 1.83 Clippy lints

* Cargo fmt
2024-11-29 02:23:54 +00:00
Age Manning
8cebc87d95 Update to latest discovery (#6486)
* Upgrade discv5 to v0.8

* Rename some logs

* Improve the NAT reporting with new discv5 metrics

* Merge branch 'unstable' into discv5-v8

* Limited Cargo.lock update

* Update yanked futures-* crates
2024-11-18 22:52:23 +00:00
Michael Sproul
9fdd53df56 Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states

* Remove unused config args

* Add comments

* Remove unwraps

* Subjective more clear implementation

* Clean up hdiff

* Update xdelta3

* Tree states archive metrics (#6040)

* Add store cache size metrics

* Add compress timer metrics

* Add diff apply compute timer metrics

* Add diff buffer cache hit metrics

* Add hdiff buffer load times

* Add blocks replayed metric

* Move metrics to store

* Future proof some metrics

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Port and clean up forwards iterator changes

* Add and polish hierarchy-config flag

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Cleaner errors

* Fix beacon_chain test compilation

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Patch a few more freezer block roots

* Fix genesis block root bug

* Fix test failing due to pending updates

* Beacon chain tests passing

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix doc lint

* Implement DB schema upgrade for hierarchical state diffs (#6193)

* DB upgrade

* Add flag

* Delete RestorePointHash

* Update docs

* Update docs

* Implement hierarchical state diffs config migration (#6245)

* Implement hierarchical state diffs config migration

* Review PR

* Remove TODO

* Set CURRENT_SCHEMA_VERSION correctly

* Fix genesis state loading

* Re-delete some PartialBeaconState stuff

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix test compilation

* Update schema downgrade test

* Fix tests

* Fix null anchor migration

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix tree states upgrade migration (#6328)

* Towards crash safety

* Fix compilation

* Move cold summaries and state roots to new columns

* Rename StateRoots chunked field

* Update prune states

* Clean hdiff CLI flag and metrics

* Fix "staged reconstruction"

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix alloy issues

* Fix staged reconstruction logic

* Prevent weird slot drift

* Remove "allow" flag

* Update CLI help

* Remove FIXME about downgrade

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Remove some unnecessary error variants

* Fix new test

* Tree states archive - review comments and metrics (#6386)

* Review PR comments and metrics

* Comments

* Add anchor metrics

* drop prev comment

* Update metadata.rs

* Apply suggestions from code review

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/store/src/hot_cold_store.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Clarify comment and remove anchor_slot garbage

* Simplify database anchor (#6397)

* Simplify database anchor

* Update beacon_node/store/src/reconstruct.rs

* Add migration for anchor

* Fix and simplify light_client store tests

* Fix incompatible config test

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* More metrics

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* New historic state cache (#6475)

* New historic state cache

* Add more metrics

* State cache hit rate metrics

* Fix store metrics

* More logs and metrics

* Fix logger

* Ensure cached states have built caches :O

* Replay blocks in preference to diffing

* Two separate caches

* Distribute cache build time to next slot

* Re-plumb historic-state-cache flag

* Clean up metrics

* Update book

* Update beacon_node/store/src/hdiff.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update beacon_node/store/src/historic_state_cache.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update database docs

* Update diagram

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Update lockbud to work with bindgen/etc

* Correct pkg name for Debian

* Remove vestigial epochs_per_state_diff

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Markdown lint

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Address Jimmy's review comments

* Simplify ReplayFrom case

* Fix and document genesis_state_root

* Typo

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into tree-states-archive

* Compute diff of validators list manually (#6556)

* Split hdiff computation

* Dedicated logic for historical roots and summaries

* Benchmark against real states

* Mutated source?

* Version the hdiff

* Add lighthouse DB config for hierarchy exponents

* Tidy up hierarchy exponents flag

* Apply suggestions from code review

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Address PR review

* Remove hardcoded paths in benchmarks

* Delete unused function in benches

* lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Test hdiff binary format stability (#6585)

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Add deprecation warning for SPRP

* Update xdelta to get rid of duplicate deps

* Document test
2024-11-18 01:51:44 +00:00
chonghe
fdf456f398 Validator manager commands for the Keymanager APIs (#6261)
* Validator manager commands for standard key-manager APIs

* Merge latest unstable

* Fix Some in lib.rs

* Replace Arg::with_name with Arg::new

* Update takes_value

* Remove clap::App

* Change App to Command

* Add command in use

* Remove generic in ArgMatches

* Fix matches.get_flag

* Fixes

* fix error handling

* SetTrue in import

* Fix

* Fix builder-proposal flag (will delete the flag later)

* Minor fix

* Fix prefer_builder_proposals

* Remove unwrap

* Error handling from Michael

* Add cli help text

* Use None in import to simplify

* Delete unwrap

* Revert flags option

* Simplify help command code

* Remove flag header in move

* Merge remote-tracking branch 'origin/unstable' into pahor/validator-manager-standard-keystore

* Add log in VC when keystore is deleted

* Delete duplicated log when validator does not exist

* Simplify log code

* Rename remove to delete

* cargo-fmt

* Try to remove a function

* make-cli

* Error handling

* Merge branch 'vm' of https://github.com/chong-he/lighthouse into vm

* Update CLI hel text

* make-cli

* Fix checks

* Merge branch 'vm' of https://github.com/chong-he/lighthouse into vm

* Try to fix check errors

* Fix test

* Remove changes

* Update flag name

* CLI display order

* Move builde_proposals flag

* Add doc

* mdlint

* Update validator_manager/src/list_validators.rs

Co-authored-by: Mac L <mjladson@pm.me>

* Delete empty line

* Fix list

* Simplify delete

* Add support to delete more validators

* Fix test

* Rename response

* Add (s)

* Add test to delete multiple validators

* Book and cli

* Make cli

* Only log when keystore is deleted

* Revise deletion log

* Add validator pubkey to error message

* Merge import

* Thank you Mac

* Test

* Add flags

* Error handling for password

* make cli

* Merge remote-tracking branch 'origin/unstable' into vm

* make cli

* Fix test

* Merge branch 'vm' of https://github.com/chong-he/lighthouse into vm

* Fix test

* vm test

* Debug trait thank you Michael

* Fix test

* Merge branch 'unstable' into vm

* test

* testing

* Combine import validator(s)

* make cli

* Add requires

* Update book

* mdlint

* Only show import log when import is successful

* delete testing

* Test for standard format

* Test standard format

* Test

* fix builder_proposals flag

* Fix test for standard format

* Add requires

* Fix vm test

* make cli

* Remove flag header

* Merge branch 'vm' of https://github.com/chong-he/lighthouse into vm

* make cli

* Delete space

* Merge branch 'vm' of https://github.com/chong-he/lighthouse into vm

* Merge branch 'unstable' into vm

* Rename delete_validator to delete_validators

* Rearrange

* Remove pub in run function

* Fix grammar

* Apply suggestions from code review

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Remove description

* Merge branch 'vm' of https://github.com/chong-he/lighthouse into vm

* Close bracket

* make cli

* Revise list code and test

* Revise import flag

* make cli

* Comment out test

* Update vm test

* Simplify

* Merge remote-tracking branch 'origin/unstable' into vm

* make cli

* Add test

* Add password as a requirement for keystore file

* Correct flags in docs

* typo
2024-10-29 04:14:06 +00:00
Age Manning
e31ac508d4 Modularize tracing executor and metrics rename (#6424)
* Tracing executor and metrics rename

* Appease clippy

* Merge branch 'unstable' into modularise-task-executor
2024-10-28 09:41:45 +00:00
Michael Sproul
a732a87846 Remove TTD flags and safe-slots-to-import-* (#6489)
* Delete SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY

* Update fork choice tests

* Remove TTD related flags

* Add deprecation warning

* Remove more dead code

* Delete EF on_merge_block tests

* Remove even more dead code

* Address Mac's review comments
2024-10-21 01:28:55 +00:00
Jimmy Chen
f3a5e256da Implement Subnet Sampling for PeerDAS (#6410)
* Add `SAMPLES_PER_SLOT` config.

* Rename `sampling` module to `peer_sampling`

* Implement subnet sampling.

* Update lookup test.

* Merge branch 'unstable' into subnet-sampling

* Merge branch 'unstable' into subnet-sampling

# Conflicts:
#	beacon_node/beacon_chain/src/data_availability_checker.rs
#	beacon_node/http_api/src/publish_blocks.rs
#	beacon_node/lighthouse_network/src/types/globals.rs
#	beacon_node/network/src/sync/manager.rs

* Merge branch 'unstable' into subnet-sampling
2024-10-04 00:27:30 +00:00
Mac L
f870b66f49 Rework Validator Client fallback mechanism (#4393)
* Rework Validator Client fallback mechanism

* Add CI workflow for fallback simulator

* Tie-break with sync distance for non-synced nodes

* Fix simulator

* Cleanup unused code

* More improvements

* Add IsOptimistic enum for readability

* Use configurable sync distance tiers

* Fix tests

* Combine status and health and improve logging

* Fix nodes not being marked as available

* Fix simulator

* Fix tests again

* Increase fallback simulator tolerance

* Add http api endpoint

* Fix todos and tests

* Update simulator

* Merge branch 'unstable' into vc-fallback

* Add suggestions

* Add id to ui endpoint

* Remove unnecessary clones

* Formatting

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Fix flag tests

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Fix conflicts

* Merge branch 'unstable' into vc-fallback

* Remove unnecessary pubs

* Simplify `compute_distance_tier` and reduce notifier awaits

* Use the more descriptive `user_index` instead of `id`

* Combine sync distance tolerance flags into one

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* wip

* Use new simulator from unstable

* Fix cli text

* Remove leftover files

* Remove old commented code

* Merge branch 'unstable' into vc-fallback

* Update cli text

* Silence candidate errors when pre-genesis

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Retry on failure

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Remove disable_run_on_all

* Remove unused error variant

* Fix out of date comment

* Merge branch 'unstable' into vc-fallback

* Remove unnecessary as_u64

* Remove more out of date comments

* Use tokio RwLock and remove parking_lot

* Merge branch 'unstable' into vc-fallback

* Formatting

* Ensure nodes are still added to total when not available

* Allow VC to detect when BN comes online

* Fix ui endpoint

* Don't have block_service as an Option

* Merge branch 'unstable' into vc-fallback

* Clean up lifetimes and futures

* Revert "Don't have block_service as an Option"

This reverts commit b5445a09e9.

* Merge branch 'unstable' into vc-fallback

* Merge branch 'unstable' into vc-fallback

* Improve rwlock sanitation using clones

* Merge branch 'unstable' into vc-fallback

* Drop read lock immediately by cloning the vec.
2024-10-03 05:57:12 +00:00
Eitan Seri-Levi
f6d46fd6e9 Fix rolling file appender bug (#6266)
* ensure file path passed to the rolling file appender only contains directories

* handle edge case

* handle edge case

* fix based on feedback

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rolling-file-apender-bug

* fmt

* linting

* ensure only bn inits tracing logfile

* ensure only bn inits tracing logfile

* Merge branch 'unstable' into rolling-file-apender-bug

* Get the metadata of `tracing_log_path` instead of `p`, which is a part of the path

* Merge pull request #11 from ackintosh/rolling-file-apender-bug-ackintosh

Get the metadata of `tracing_log_path` instead of `p`

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rolling-file-apender-bug

* fmt
2024-10-03 02:00:56 +00:00
Eitan Seri-Levi
82faf975b3 Add {fork_name}_enabled functions (#5951)
* add fork_name_enabled fn to Forkname impl

* refactor codebase to use new fork_enabled fn

* fmt

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fork-ord-impl

* small code cleanup

* resolve merge conflicts

* fix beacon chain test

* merge conflicts

* fix ef test issue

* resolve merge conflicts
2024-10-03 02:00:52 +00:00
Michael Sproul
2792705331 Lenient duplicate checks on HTTP API for block publication (#5574)
* start splitting gossip verification

* WIP

* Gossip verify separate (#7)

* save

* save

* make ProvenancedBlock concrete

* delete into gossip verified block contents

* get rid of IntoBlobSidecar trait

* remove IntoGossipVerified trait

* get tests compiling

* don't check sidecar slashability in publish

* remove second publish closure

* drop blob bool. also prefer using message index over index of position in list

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Fix low-hanging tests

* Fix tests and clean up

* Clean up imports

* more cleanup

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Further refine behaviour and add tests

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Remove empty line

* Fix test (block is not fully imported just gossip verified)

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Update for unstable & use empty blob list

* Update comment

* Add test for duplicate block case

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Clarify unreachable case

* Fix another publish_block case

* Remove unreachable case in filter chain segment

* Revert unrelated blob optimisation

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Fix merge conflicts

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Fix some compilation issues. Impl is fucked though

* Support peerDAS

* Fix tests

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Fix conflict

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate

* Address review comments

* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
2024-09-24 04:52:44 +00:00
Michael Sproul
1447eeb40b Improve single-node testnet support and Arc NetworkConfig/ChainSpec (#6396)
* Arc ChainSpec and NetworkConfig

* Fix release tests

* Fix lint

* Merge remote-tracking branch 'origin/unstable' into single-node-testnet
2024-09-24 00:16:18 +00:00
Eitan Seri-Levi
b619f1ab5c Make BeaconChain::kzg field mandatory (#6267)
* make kzg field required

* update todo

* always load trusted setup WIP

* fmt

* use new rust_eth_kzg version

* merge conlficts

* add kzg fn with trusted setup disabled

* as_slice

* add kzg with no precomp

* ignore udep for kzg

* refactor kzg init

* fix peerdas kzg schedule

* fix

* udeps

* uuuudeps

* merge conflict resolved

* merge conflict

* merge conflicts

* resolve TODO

* update

* move kzg to a test util fn

* remove trusted setup default impl

* lint fmt

* fix failing test

* lint

* fix test

* Merge branch 'unstable' into beacon-chain-kzg-field-required
2024-09-23 04:54:32 +00:00