Adds the required boilerplate code for the Gloas (Glamsterdam) hard fork. This allows PRs testing Gloas-candidate features to test fork transition.
This also includes de-duplication of post-Bellatrix readiness notifiers from #6797 (credit to @dapplion)
This PR fixes a bug where wrong columns could get processed immediately after a CGC increase.
Scenario:
- The node's CGC increased due to additional validators attached to it (lets say from 10 to 11)
- The new CGC is advertised and new subnets are subscribed immediately, however the change won't be effective in the data availability check until the next epoch (See [this](ab0e8870b4/beacon_node/beacon_chain/src/validator_custody.rs (L93-L99))). Data availability checker still only require 10 columns for the current epoch.
- During this time, data columns for the additional custody column (lets say column 11) may arrive via gossip as we're already subscribed to the topic, and it may be incorrectly used to satisfy the existing data availability requirement (10 columns), and result in this additional column (instead of a required one) getting persisted, resulting in database inconsistency.
Fix Clippy for recently released Rust 1.90 beta. There may be more changes required when Rust 1.89 stable is released in a few days, but possibly not 🤞
I noticed that we are serving preset values for Fulu on mainnet nodes prior to the fork. This has already gone live in v7.1.0, but should hopefully be handled in a graceful way by API consumers.
This PR _reverts_ the serving of Fulu data prior to Fulu, by serving Fulu data only if Fulu is scheduled.
Closes#6855
Add PeerDAS broadcast validation tests and fix a small bug where `sampling_columns_indices` is none (indicating that we've already sampled the necessary columns) and `process_gossip_data_columns` gets called
N/A
After the electra fork which includes EIP 6110, the beacon node no longer needs the eth1 bridging mechanism to include new deposits as they are provided by the EL as a `deposit_request`. So after electra + a transition period where the finalized bridge deposits pre-fork are included through the old mechanism, we no longer need the elaborate machinery we had to get deposit contract data from the execution layer.
Since holesky has already forked to electra and completed the transition period, this PR basically checks to see if removing all the eth1 related logic leads to any surprises.
Partially https://github.com/sigp/lighthouse/issues/6291
This PR removes the reprocess event channel from being externally exposed. All work events are now sent through the single `BeaconProcessorSend` channel. I've introduced a new `Work::Reprocess` enum variant which we then use to schedule jobs for reprocess. I've also created a new scheduler module which will eventually house the different scheduler impls.
This is all needed as an initial step to generalize the beacon processor
A "full" implementation for the generalized beacon processor can be found here
https://github.com/sigp/lighthouse/pull/6448
I'm going to try to break up the full implementation into smaller PR's so it can actually be reviewed
#6970
This allows for us to receive `SingleAttestation` over gossip and process it without converting. There is still a conversion to `Attestation` as a final step in the attestation verification process, but by then the `SingleAttestation` is fully verified.
I've also fully removed the `submitPoolAttestationsV1` endpoint as its been deprecated
I've also pre-emptively deprecated supporting `Attestation` in `submitPoolAttestationsV2` endpoint. See here for more info: https://github.com/ethereum/beacon-APIs/pull/531
I tried to the minimize the diff here by only making the "required" changes. There are some unnecessary complexities with the way we manage the different attestation verification wrapper types. We could probably consolidate this to one wrapper type and refactor this even further. We could leave that to a separate PR if we feel like cleaning things up in the future.
Note that I've also updated the test harness to always submit `SingleAttestation` regardless of fork variant. I don't see a problem in that approach and it allows us to delete more code :)
Resolves#6767
This PR implements a basic version of validator custody.
- It introduces a new `CustodyContext` object which contains info regarding number of validators attached to a node and the custody count they contribute to the cgc.
- The `CustodyContext` is added in the da_checker and has methods for returning the current cgc and the number of columns to sample at head. Note that the logic for returning the cgc existed previously in the network globals.
- To estimate the number of validators attached, we use the `beacon_committee_subscriptions` endpoint. This might overestimate the number of validators actually publishing attestations from the node in the case of multi BN setups. We could also potentially use the `publish_attestations` endpoint to get a more conservative estimate at a later point.
- Anytime there's a change in the `custody_group_count` due to addition/removal of validators, the custody context should send an event on a broadcast channnel. The only subscriber for the channel exists in the network service which simply subscribes to more subnets. There can be additional subscribers in sync that will start a backfill once the cgc changes.
TODO
- [ ] **NOT REQUIRED:** Currently, the logic only handles an increase in validator count and does not handle a decrease. We should ideally unsubscribe from subnets when the cgc has decreased.
- [ ] **NOT REQUIRED:** Add a service in the `CustodyContext` that emits an event once `MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS ` passes after updating the current cgc. This event should be picked up by a subscriber which updates the enr and metadata.
- [x] Add more tests
The endpoint `/eth/v1/beacon/states/head/validator_balances` returns an empty data when the data field is `[]`. According to the beacon API spec, it should return the balances of all validators:
Reference: https://ethereum.github.io/beacon-APIs/#/Beacon/postStateValidatorBalances
`If the supplied list is empty (i.e. the body is []) or no body is supplied then balances will be returned for all validators.`
This PR changes so that: `curl -X 'POST' 'http://localhost:5052/eth/v1/beacon/states/head/validator_balances' -d '[]' | jq` returns balances of all validators.
Resolves https://github.com/sigp/lighthouse/issues/7414
The health endpoint returns a 503 if the engine state is offline. The default state for the engine is `Offline`. So until the first request to the EL is made and the state is updated, the health endpoint will keep returning 503s.
This PR changes the default state to Online to avoid that. I don't think this causes any issues because in case the EL is actually offline, the first fcu will set the state to offline.
Pending testing on kurtosis.
Did not find a specific issue beside https://github.com/sigp/lighthouse/issues/6821
Leverage `whistleblower_reward_quotient_for_state` to have accurate post-electra `proposer_slashings` and `attester_slashings` fields returned by `/eth/v1/beacon/rewards/blocks/<id>`.
#7294
Fix the filtering logic so that we actually filter by committee index for both `Base` and `Electra` attestations.
Added a tiny optimization when calculating committee_index to prevent unneeded memory allocations
Added a regression test
N/A
Adds endpoints to add and remove trusted peers from the http api. The added peers are trusted peers so they won't be disconnected for bad scores. We try to maintain a connection to the peer in case they disconnect from us by trying to dial it every heartbeat.
- Part of https://github.com/sigp/lighthouse/issues/6767
Validator custody makes the CGC and set of sampling columns dynamic. Right now this information is stored twice:
- in the data availability checker
- in the network globals
If that state becomes dynamic we must make sure it is in sync updating it twice, or guarding it behind a mutex. However, I noted that we don't really have to keep the CGC inside the data availability checker. All consumers can actually read it from the network globals, and we can update `make_available` to read the expected count of data columns from the block.
- #6452 (partially)
Remove dependencies on `store` and `lighthouse_network` from `eth2`. This was achieved as follows:
- depend on `enr` and `multiaddr` directly instead of using `lighthouse_network`'s reexports.
- make `lighthouse_network` responsible for converting between API and internal types.
- in two cases, remove complex internal types and use the generic `serde_json::Value` instead - this is not ideal, but should be fine for now, as this affects two internal non-spec endpoints which are meant for debugging, unstable, and subject to change without notice anyway. Inspired by #6679. The alternative is to move all relevant types to `eth2` or `types` instead - what do you think?
N/A
2 changes:
1. Replace Option::map_or(true, ...) with is_none_or(...)
2. Remove unnecessary `Into::into` blocks where the type conversion is apparent from the types
Closes#6983
`GET v2/validator/aggregate_attestation` is not backwards compatible. It only works for post electra attestations. This PR adds backwards compatibility and additional test coverage. We should include this in the upcoming 7.0 beta release if possible
Closes:
- https://github.com/sigp/lighthouse/issues/6818
Use `MAX_EFFECTIVE_BALANCE_ELECTRA` (2048) for attestation reward calculations involving Electra.
Add a new `InteropGenesisBuilder` that tries to provide a more flexible way to build genesis states. Unfortunately due to lifetime jank, it is quite unergonomic at present. We may want to refactor this builder in future to make it easier to use.
Addresses #6706
This PR activates PeerDAS at the Fulu fork epoch instead of `EIP_7594_FORK_EPOCH`. This means we no longer support testing PeerDAS with Deneb / Electrs, as it's now part of a hard fork.
* First pass
* Add restrictions to RuntimeVariableList api
* Use empty_uninitialized and fix warnings
* Fix some todos
* Merge branch 'unstable' into max-blobs-preset
* Fix take impl on RuntimeFixedList
* cleanup
* Fix test compilations
* Fix some more tests
* Fix test from unstable
* Merge branch 'unstable' into max-blobs-preset
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Remove footgun function
* Minor simplifications
* Move from preset to config
* Fix typo
* Revert "Remove footgun function"
This reverts commit de01f923c7.
* Try fixing tests
* Thread through ChainSpec
* Fix release tests
* Move RuntimeFixedVector into module and rename
* Add test
* Remove empty RuntimeVarList awefullness
* Fix tests
* Simplify BlobSidecarListFromRoot
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Bump quota to account for new target (6)
* Remove clone
* Fix issue from review
* Try to remove ugliness
* Merge branch 'unstable' into max-blobs-preset
* Fix max value
* Fix doctest
* Fix formatting
* Fix max check
* Delete hardcoded max_blobs_per_block in RPC limits
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Get blobs from EL.
Co-authored-by: Michael Sproul <michael@sigmaprime.io>
* Avoid cloning blobs after fetching blobs.
* Address review comments and refactor code.
* Fix lint.
* Move blob computation metric to the right spot.
* Merge branch 'unstable' into das-fetch-blobs
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
# beacon_node/beacon_chain/src/block_verification.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
* Gradual publication of data columns for supernodes.
* Recompute head after importing block with blobs from the EL.
* Fix lint
* Merge branch 'unstable' into das-fetch-blobs
* Use blocking task instead of async when computing cells.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs
* Fix semantic conflicts
* Downgrade error log.
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/execution_layer/src/engine_api.rs
# beacon_node/execution_layer/src/engine_api/json_structures.rs
# beacon_node/network/src/network_beacon_processor/gossip_methods.rs
# beacon_node/network/src/network_beacon_processor/mod.rs
# beacon_node/network/src/network_beacon_processor/sync_methods.rs
* Merge branch 'unstable' into das-fetch-blobs
* Publish block without waiting for blob and column proof computation.
* Address review comments and refactor.
* Merge branch 'unstable' into das-fetch-blobs
* Fix test and docs.
* Comment cleanups.
* Merge branch 'unstable' into das-fetch-blobs
* Address review comments and cleanup
* Address review comments and cleanup
* Refactor to de-duplicate gradual publication logic.
* Add more logging.
* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs
# Conflicts:
# Cargo.lock
* Fix incorrect comparison on `num_fetched_blobs`.
* Implement gradual blob publication.
* Merge branch 'unstable' into das-fetch-blobs
* Inline `publish_fn`.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
* Gossip verify blobs before publishing
* Avoid queries for 0 blobs and error for duplicates
* Gossip verified engine blob before processing them, and use observe cache to detect duplicates before publishing.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
# Conflicts:
# beacon_node/network/src/network_beacon_processor/mod.rs
* Merge branch 'unstable' into das-fetch-blobs
* Fix invalid commitment inclusion proofs in blob sidecars created from EL blobs.
* Only publish EL blobs triggered from gossip block, and not RPC block.
* Downgrade gossip blob log to `debug`.
* Merge branch 'unstable' into das-fetch-blobs
* Merge branch 'unstable' into das-fetch-blobs
* Grammar
* start splitting gossip verification
* WIP
* Gossip verify separate (#7)
* save
* save
* make ProvenancedBlock concrete
* delete into gossip verified block contents
* get rid of IntoBlobSidecar trait
* remove IntoGossipVerified trait
* get tests compiling
* don't check sidecar slashability in publish
* remove second publish closure
* drop blob bool. also prefer using message index over index of position in list
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix low-hanging tests
* Fix tests and clean up
* Clean up imports
* more cleanup
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Further refine behaviour and add tests
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Remove empty line
* Fix test (block is not fully imported just gossip verified)
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Update for unstable & use empty blob list
* Update comment
* Add test for duplicate block case
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Clarify unreachable case
* Fix another publish_block case
* Remove unreachable case in filter chain segment
* Revert unrelated blob optimisation
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix merge conflicts
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix some compilation issues. Impl is fucked though
* Support peerDAS
* Fix tests
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix conflict
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Address review comments
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* persist light client updates
* update beacon chain to serve light client updates
* resolve todos
* cache best update
* extend cache parts
* is better light client update
* resolve merge conflict
* initial api changes
* add lc update db column
* fmt
* added tests
* add sim
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* fix some weird issues with the simulator
* tests
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* test changes
* merge conflict
* testing
* started work on ef tests and some code clean up
* update tests
* linting
* noop pre altair, were still failing on electra though
* allow for zeroed light client header
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* merge unstable
* remove unwraps
* remove unwraps
* fetch bootstrap without always querying for state
* storing bootstrap parts in db
* mroe code cleanup
* test
* prune sync committee branches from dropped chains
* Update light_client_update.rs
* merge unstable
* move functionality to helper methods
* refactor is best update fn
* refactor is best update fn
* improve organization of light client server cache logic
* fork diget calc, and only spawn as many blcoks as we need for the lc update test
* resovle merge conflict
* add electra bootstrap logic, add logic to cache current sync committee
* add latest sync committe branch cache
* fetch lc update from the cache if it exists
* fmt
* Fix beacon_chain tests
* Add debug code to update ranking_order ef test
* Fix compare code
* merge conflicts
* merge conflict
* add better error messaging
* resolve merge conflicts
* remove lc update from basicsim
* rename sync comittte variable and fix persist condition
* refactor get_light_client_update logic
* add better comments, return helpful error messages over http and rpc
* pruning canonical non checkpoint slots
* fix test
* rerun test
* update pruning logic, add tests
* fix tests
* fix imports
* fmt
* refactor db code
* Refactor db method
* Refactor db method
* add additional comments
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-bootstrap
* fix merge
* linting
* merge conflict
* prevent overflow
* enable lc server for http api tests
* fix tests
* remove prints
* remove warning
* revert change
