mirror of
https://github.com/sigp/lighthouse.git
synced 2026-03-03 00:31:50 +00:00
09065a851fb33c13b5ba3e2889b22daf0020ced4
41 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Eitan Seri-Levi
|
f67084a571 |
Remove reprocess channel (#7437)
Partially https://github.com/sigp/lighthouse/issues/6291 This PR removes the reprocess event channel from being externally exposed. All work events are now sent through the single `BeaconProcessorSend` channel. I've introduced a new `Work::Reprocess` enum variant which we then use to schedule jobs for reprocess. I've also created a new scheduler module which will eventually house the different scheduler impls. This is all needed as an initial step to generalize the beacon processor A "full" implementation for the generalized beacon processor can be found here https://github.com/sigp/lighthouse/pull/6448 I'm going to try to break up the full implementation into smaller PR's so it can actually be reviewed |
||
|
Eitan Seri-Levi
|
6786b9d12a |
Single attestation "Full" implementation (#7444)
#6970 This allows for us to receive `SingleAttestation` over gossip and process it without converting. There is still a conversion to `Attestation` as a final step in the attestation verification process, but by then the `SingleAttestation` is fully verified. I've also fully removed the `submitPoolAttestationsV1` endpoint as its been deprecated I've also pre-emptively deprecated supporting `Attestation` in `submitPoolAttestationsV2` endpoint. See here for more info: https://github.com/ethereum/beacon-APIs/pull/531 I tried to the minimize the diff here by only making the "required" changes. There are some unnecessary complexities with the way we manage the different attestation verification wrapper types. We could probably consolidate this to one wrapper type and refactor this even further. We could leave that to a separate PR if we feel like cleaning things up in the future. Note that I've also updated the test harness to always submit `SingleAttestation` regardless of fork variant. I don't see a problem in that approach and it allows us to delete more code :) |
||
|
ethDreamer
|
7684d1f866 |
ContextDeserialize and Beacon API Improvements (#7372)
* #7286 * BeaconAPI is not returning a versioned response when it should for some V1 endpoints * these [strange functions with vX in the name that still accept `endpoint_version` arguments](https://github.com/sigp/lighthouse/blob/stable/beacon_node/http_api/src/produce_block.rs#L192) This refactor is a prerequisite to get the fulu EF tests running. |
||
|
Eitan Seri-Levi
|
276eda3dfe |
POST /eth/v2/beacon/pool/attestations bugfixes (#6867)
|
||
|
Eitan Seri-Levi
|
06329ec2d1 |
SingleAttestation implementation (#6488)
* First pass * Add restrictions to RuntimeVariableList api * Use empty_uninitialized and fix warnings * Fix some todos * Merge branch 'unstable' into max-blobs-preset * Fix take impl on RuntimeFixedList * cleanup * Fix test compilations * Fix some more tests * Fix test from unstable * Merge branch 'unstable' into max-blobs-preset * SingleAttestation * Add post attestation v2 endpoint logic to attestation service * Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation * Implement "Bugfix and more withdrawal tests" * Implement "Add missed exit checks to consolidation processing" * Implement "Update initial earliest_exit_epoch calculation" * Implement "Limit consolidating balance by validator.effective_balance" * Implement "Use 16-bit random value in validator filter" * Implement "Do not change creds type on consolidation" * some tests and fixed attestqtion calc * Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation * Rename PendingPartialWithdraw index field to validator_index * Skip slots to get test to pass and add TODO * Implement "Synchronously check all transactions to have non-zero length" * Merge remote-tracking branch 'origin/unstable' into max-blobs-preset * Remove footgun function * Minor simplifications * Move from preset to config * Fix typo * Revert "Remove footgun function" This reverts commit |
||
|
Pawan Dhananjay
|
1f6850fae2 |
Rust 1.84 lints (#6781)
* Fix few lints * Fix remaining lints * Use fully qualified syntax |
||
|
Daniel Knopik
|
02cb2d68ff |
Enable lints for tests only running optimized (#6664)
* enable linting optimized-only tests * fix automatically fixable or obvious lints * fix suspicious_open_options by removing manual options * fix `await_holding_lock`s * avoid failing lint due to now disabled `#[cfg(debug_assertions)]` * reduce future sizes in tests * fix accidently flipped assert logic * restore holding lock for web3signer download * Merge branch 'unstable' into lint-opt-tests |
||
|
ethDreamer
|
86891e6d0f |
builder gas limit & some refactoring (#6583)
* Cache gas_limit * Payload Parameters Refactor * Enforce Proposer Gas Limit * Fixed and Added New Tests * Fix Beacon Chain Tests |
||
|
Michael Sproul
|
2792705331 |
Lenient duplicate checks on HTTP API for block publication (#5574)
* start splitting gossip verification * WIP * Gossip verify separate (#7) * save * save * make ProvenancedBlock concrete * delete into gossip verified block contents * get rid of IntoBlobSidecar trait * remove IntoGossipVerified trait * get tests compiling * don't check sidecar slashability in publish * remove second publish closure * drop blob bool. also prefer using message index over index of position in list * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Fix low-hanging tests * Fix tests and clean up * Clean up imports * more cleanup * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Further refine behaviour and add tests * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Remove empty line * Fix test (block is not fully imported just gossip verified) * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Update for unstable & use empty blob list * Update comment * Add test for duplicate block case * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Clarify unreachable case * Fix another publish_block case * Remove unreachable case in filter chain segment * Revert unrelated blob optimisation * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Fix merge conflicts * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Fix some compilation issues. Impl is fucked though * Support peerDAS * Fix tests * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Fix conflict * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate * Address review comments * Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate |
||
|
Eitan Seri-Levi
|
a94b12b4d5 |
Persist light client bootstrap (#5915)
* persist light client updates * update beacon chain to serve light client updates * resolve todos * cache best update * extend cache parts * is better light client update * resolve merge conflict * initial api changes * add lc update db column * fmt * added tests * add sim * Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates * fix some weird issues with the simulator * tests * Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates * test changes * merge conflict * testing * started work on ef tests and some code clean up * update tests * linting * noop pre altair, were still failing on electra though * allow for zeroed light client header * Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates * merge unstable * remove unwraps * remove unwraps * fetch bootstrap without always querying for state * storing bootstrap parts in db * mroe code cleanup * test * prune sync committee branches from dropped chains * Update light_client_update.rs * merge unstable * move functionality to helper methods * refactor is best update fn * refactor is best update fn * improve organization of light client server cache logic * fork diget calc, and only spawn as many blcoks as we need for the lc update test * resovle merge conflict * add electra bootstrap logic, add logic to cache current sync committee * add latest sync committe branch cache * fetch lc update from the cache if it exists * fmt * Fix beacon_chain tests * Add debug code to update ranking_order ef test * Fix compare code * merge conflicts * merge conflict * add better error messaging * resolve merge conflicts * remove lc update from basicsim * rename sync comittte variable and fix persist condition * refactor get_light_client_update logic * add better comments, return helpful error messages over http and rpc * pruning canonical non checkpoint slots * fix test * rerun test * update pruning logic, add tests * fix tests * fix imports * fmt * refactor db code * Refactor db method * Refactor db method * add additional comments * Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-bootstrap * fix merge * linting * merge conflict * prevent overflow * enable lc server for http api tests * fix tests * remove prints * remove warning * revert change |
||
|
Eitan Seri-Levi
|
99e53b88c3 |
Migrate from ethereum-types to alloy-primitives (#6078)
* Remove use of ethers_core::RlpStream
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core
* Remove old code
* Simplify keccak call
* Remove unused package
* Merge branch 'unstable' of https://github.com/ethDreamer/lighthouse into remove_use_of_ethers_core
* Merge branch 'unstable' into remove_use_of_ethers_core
* Run clippy
* Merge branch 'remove_use_of_ethers_core' of https://github.com/dospore/lighthouse into remove_use_of_ethers_core
* Check all cargo fmt
* migrate to alloy primitives init
* fix deps
* integrate alloy-primitives
* resolve dep issues
* more changes based on dep changes
* add TODOs
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into remove_use_of_ethers_core
* Revert lock
* Add BeaconBlocksByRange v3
* continue migration
* Revert "Add BeaconBlocksByRange v3"
This reverts commit
|
||
|
realbigsean
|
13ffdd211d |
Beacon api + validator electra (#5744)
* Attestation superstruct changes for EIP 7549 (#5644)
* update
* experiment
* superstruct changes
* revert
* superstruct changes
* fix tests
* indexed attestation
* indexed attestation superstruct
* updated TODOs
* `superstruct` the `AttesterSlashing` (#5636)
* `superstruct` Attester Fork Variants
* Push a little further
* Deal with Encode / Decode of AttesterSlashing
* not so sure about this..
* Stop Encode/Decode Bounds from Propagating Out
* Tons of Changes..
* More Conversions to AttestationRef
* Add AsReference trait (#15)
* Add AsReference trait
* Fix some snafus
* Got it Compiling! :D
* Got Tests Building
* Get beacon chain tests compiling
---------
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes
* Make EF Tests Fork-Agnostic (#5713)
* Finish EF Test Fork Agnostic (#5714)
* Superstruct `AggregateAndProof` (#5715)
* Upgrade `superstruct` to `0.8.0`
* superstruct `AggregateAndProof`
* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes
* cargo fmt
* Merge pull request #5726 from realbigsean/electra_attestation_changes
Merge unstable into Electra attestation changes
* process withdrawals updates
* cleanup withdrawals processing
* update `process_operations` deposit length check
* add apply_deposit changes
* add execution layer withdrawal request processing
* process deposit receipts
* add consolidation processing
* update process operations function
* exit updates
* clean up
* update slash_validator
* EIP7549 `get_attestation_indices` (#5657)
* get attesting indices electra impl
* fmt
* get tests to pass
* fmt
* fix some beacon chain tests
* fmt
* fix slasher test
* fmt got me again
* fix more tests
* fix tests
* Some small changes (#5739)
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* cargo fmt (#5740)
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* fix attestation verification
* Add new engine api methods
* Fix the versioning of v4 requests
* Handle new engine api methods in mock EL
* Note todo
* Fix todos
* Add support for electra fields in getPayloadBodies
* Add comments for potential versioning confusion
* udpates for aggregate attestation endpoint
* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra
* Sketch op pool changes
* fix get attesting indices (#5742)
* fix get attesting indices
* better errors
* fix compile
* only get committee index once
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* Ef test fixes (#5753)
* attestation related ef test fixes
* delete commented out stuff
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* Merge branch 'block-processing-electra' of https://github.com/sigp/lighthouse into electra-engine-api
* Merge branch 'electra-engine-api' of https://github.com/sigp/lighthouse into beacon-api-electra
* Fix Aggregation Pool for Electra (#5754)
* Fix Aggregation Pool for Electra
* Remove Outdated Interface
* fix ssz (#5755)
* Get `electra_op_pool` up to date (#5756)
* fix get attesting indices (#5742)
* fix get attesting indices
* better errors
* fix compile
* only get committee index once
* Ef test fixes (#5753)
* attestation related ef test fixes
* delete commented out stuff
* Fix Aggregation Pool for Electra (#5754)
* Fix Aggregation Pool for Electra
* Remove Outdated Interface
* fix ssz (#5755)
---------
Co-authored-by: realbigsean <sean@sigmaprime.io>
* Revert "Get `electra_op_pool` up to date (#5756)" (#5757)
This reverts commit
|
||
|
Michael Sproul
|
2f0af2be89 |
Unimplement TreeHash for BeaconState (#6083)
* Unimplement `TreeHash` for `BeaconState` |
||
|
realbigsean
|
858b01f4e3 |
Block processing electra (#5741)
* Attestation superstruct changes for EIP 7549 (#5644)
* update
* experiment
* superstruct changes
* revert
* superstruct changes
* fix tests
* indexed attestation
* indexed attestation superstruct
* updated TODOs
* `superstruct` the `AttesterSlashing` (#5636)
* `superstruct` Attester Fork Variants
* Push a little further
* Deal with Encode / Decode of AttesterSlashing
* not so sure about this..
* Stop Encode/Decode Bounds from Propagating Out
* Tons of Changes..
* More Conversions to AttestationRef
* Add AsReference trait (#15)
* Add AsReference trait
* Fix some snafus
* Got it Compiling! :D
* Got Tests Building
* Get beacon chain tests compiling
---------
Co-authored-by: Michael Sproul <micsproul@gmail.com>
* Merge remote-tracking branch 'upstream/unstable' into electra_attestation_changes
* Make EF Tests Fork-Agnostic (#5713)
* Finish EF Test Fork Agnostic (#5714)
* Superstruct `AggregateAndProof` (#5715)
* Upgrade `superstruct` to `0.8.0`
* superstruct `AggregateAndProof`
* Merge remote-tracking branch 'sigp/unstable' into electra_attestation_changes
* cargo fmt
* Merge pull request #5726 from realbigsean/electra_attestation_changes
Merge unstable into Electra attestation changes
* process withdrawals updates
* cleanup withdrawals processing
* update `process_operations` deposit length check
* add apply_deposit changes
* add execution layer withdrawal request processing
* process deposit receipts
* add consolidation processing
* update process operations function
* exit updates
* clean up
* update slash_validator
* EIP7549 `get_attestation_indices` (#5657)
* get attesting indices electra impl
* fmt
* get tests to pass
* fmt
* fix some beacon chain tests
* fmt
* fix slasher test
* fmt got me again
* fix more tests
* fix tests
* Some small changes (#5739)
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* cargo fmt (#5740)
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* fix attestation verification
* Sketch op pool changes
* fix get attesting indices (#5742)
* fix get attesting indices
* better errors
* fix compile
* only get committee index once
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* Ef test fixes (#5753)
* attestation related ef test fixes
* delete commented out stuff
* Merge branch 'electra_attestation_changes' of https://github.com/sigp/lighthouse into block-processing-electra
* Fix Aggregation Pool for Electra (#5754)
* Fix Aggregation Pool for Electra
* Remove Outdated Interface
* fix ssz (#5755)
* Get `electra_op_pool` up to date (#5756)
* fix get attesting indices (#5742)
* fix get attesting indices
* better errors
* fix compile
* only get committee index once
* Ef test fixes (#5753)
* attestation related ef test fixes
* delete commented out stuff
* Fix Aggregation Pool for Electra (#5754)
* Fix Aggregation Pool for Electra
* Remove Outdated Interface
* fix ssz (#5755)
---------
Co-authored-by: realbigsean <sean@sigmaprime.io>
* Revert "Get `electra_op_pool` up to date (#5756)" (#5757)
This reverts commit
|
||
|
Eitan Seri-Levi
|
ee69e14db9 |
Add is_parent_strong proposer re-org check (#5417)
* initial fork choice additions * add helper fns * add is_parent_strong * Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check * disabling proposer reorg should set parent_threshold to u64 max * add new flag, is_parent_strong check in override fcu params * cherry-pick changes * Merge branch 'unstable' of https://github.com/sigp/lighthouse into add_is_parent_strong_check * cleanup * fmt * Minor review tweaks |
||
|
Michael Sproul
|
f17fb291b7 |
Handle unknown head during attestation publishing (#5010)
* Handle unknown head during attestation publishing * Merge remote-tracking branch 'origin/unstable' into queue-http-attestations * Simplify task spawner * Improve logging * Add a test * Improve error logging * Merge remote-tracking branch 'origin/unstable' into queue-http-attestations * Fix beta compiler warnings |
||
|
Michael Sproul
|
a403138ed0 |
Reduce size of futures in HTTP API to prevent stack overflows (#5104)
* Box::pin a few big futures * Arc the blocks early in publication * Fix more tests |
||
|
Eitan Seri-Levi
|
9c1505d082 |
Block v3 builder boost factor (#5035)
* builder boost factor * default boost factor * revert * deprecate always_prefer_builder_payload, builder-profit-threshold, ignore_builder_override_suggestion_threshold and builder_comparison_factor flags * revert * set deprecated flags to no op, revert should_override_builder * fix test, calc boosted relay value correctly, dont calculate if none * Add deprecation warnings and restore CLI docs |
||
|
Michael Sproul
|
af11e78ae1 |
Clean up blockv3 metadata and client (#5015)
* Improve block production v3 client * Delete wayward line * Overhaul JSON endpoint as well * Rename timeout param * Update tests * I broke everything * Ah this is an insane fix * Remove unnecessary optionals * Doc fix |
||
|
Pawan Dhananjay
|
31044402ee |
Sidecar inclusion proof (#4900)
* Refactor BlobSidecar to new type * Fix some compile errors * Gossip verification compiles * Fix http api types take 1 * Fix another round of compile errors * Beacon node crate compiles * EF tests compile * Remove all blob signing from VC * fmt * Tests compile * Fix some tests * Fix more http tests * get compiling * Fix gossip conditions and tests * Add basic proof generation and verification * remove unnecessary ssz decode * add back build_sidecar * remove default at fork for blobs * fix beacon chain tests * get relase tests compiling * fix lints * fix existing spec tests * add new ef tests * fix gossip duplicate rule * lints * add back sidecar signature check in gossip * add finalized descendant check to blob sidecar gossip * fix error conversion * fix release tests * sidecar inclusion self review cleanup * Add proof verification and computation metrics * Remove accidentally committed file * Unify some block and blob errors; add slashing conditions for sidecars * Address review comment * Clean up re-org tests (#4957) * Address more review comments * Add Comments & Eliminate Unnecessary Clones * update names * Update beacon_node/beacon_chain/src/metrics.rs Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Update beacon_node/network/src/network_beacon_processor/tests.rs Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * pr feedback * fix test compile * Sidecar Inclusion proof small refactor and updates (#4967) * Update some comments, variables and small cosmetic fixes. * Couple blobs and proofs into a tuple in `PayloadAndBlobs` for simplicity and safety. * Update function comment. * Update testing/ef_tests/src/cases/merkle_proof_validity.rs Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> * Rename the block and blob wrapper types used in the beacon API interfaces. * make sure gossip invalid blobs are passed to the slasher (#4970) * Add blob headers to slasher before adding to DA checker * Replace Vec with HashSet in BlockQueue * fmt * Rename gindex -> index * Simplify gossip condition --------- Co-authored-by: realbigsean <seananderson33@gmail.com> Co-authored-by: realbigsean <sean@sigmaprime.io> Co-authored-by: Michael Sproul <michael@sigmaprime.io> Co-authored-by: Mark Mackey <mark@sigmaprime.io> Co-authored-by: Jimmy Chen <jchen.tc@gmail.com> |
||
|
Eitan Seri-Levi
|
07f53b18fc |
Block v3 endpoint (#4629)
## Issue Addressed #4582 ## Proposed Changes Add a new v3 block fetching flow that can decide to return a Full OR Blinded payload ## Additional Info Co-authored-by: Michael Sproul <micsproul@gmail.com> |
||
|
Michael Sproul
|
8db44decb7 |
Fix parent_beacon_block_root during proposer prep (#4703)
* Fix `parent_beacon_block_root` during prep/reorg * Fix another bug and add tests * Remove overzealous payload attributes check |
||
|
realbigsean
|
33dd13c798 |
Refactor deneb block processing (#4511)
* Revert "fix merge"
This reverts commit
|
||
|
realbigsean
|
b96db45090 | Merge branch 'unstable' of https://github.com/sigp/lighthouse into merge-unstable-deneb-jul-14 | ||
|
Michael Sproul
|
6c375205fb |
Fix HTTP state API bug and add --epochs-per-migration (#4236)
## Issue Addressed
Fix an issue observed by `@zlan` on Discord where Lighthouse would sometimes return this error when looking up states via the API:
> {"code":500,"message":"UNHANDLED_ERROR: ForkChoiceError(MissingProtoArrayBlock(0xc9cf1495421b6ef3215d82253b388d77321176a1dcef0db0e71a0cd0ffc8cdb7))","stacktraces":[]}
## Proposed Changes
The error stems from a faulty assumption in the HTTP API logic: that any state in the hot database must have its block in fork choice. This isn't true because the state's hot database may update much less frequently than the fork choice store, e.g. if reconstructing states (where freezer migration pauses), or if the freezer migration runs slowly. There could also be a race between loading the hot state and checking fork choice, e.g. even if the finalization migration of DB+fork choice were atomic, the update could happen between the 1st and 2nd calls.
To address this I've changed the HTTP API logic to use the finalized block's execution status as a fallback where it is safe to do so. In the case where a block is non-canonical and prior to finalization (permanently orphaned) we default `execution_optimistic` to `true`.
## Additional Info
I've also added a new CLI flag to reduce the frequency of the finalization migration as this is useful for several purposes:
- Spacing out database writes (less frequent, larger batches)
- Keeping a limited chain history with high availability, e.g. the last month in the hot database.
This new flag made it _substantially_ easier to test this change. It was extracted from `tree-states` (where it's called `--db-migration-period`), which is why this PR also carries the `tree-states` label.
|
||
|
ethDreamer
|
c1d47da02d |
Update engine_api to latest version (#4223)
* Update Engine API to Latest * Get Mock EE Working * Fix Mock EE * Update Engine API Again * Rip out get_blobs_bundle Stuff * Fix Test Harness * Fix Clippy Complaints * Fix Beacon Chain Tests |
||
|
Pawan Dhananjay
|
b6c0e91c05 | Merge branch 'eip4844' into deneb-free-blobs | ||
|
Michael Sproul
|
b90c0c3fb1 |
Make re-org strat more cautious and add more config (#4151)
## Proposed Changes This change attempts to prevent failed re-orgs by: 1. Lowering the re-org cutoff from 2s to 1s. This is informed by a failed re-org attempted by @yorickdowne's node. The failed block was requested in the 1.5-2s window due to a Vouch failure, and failed to propagate to the majority of the network before the attestation deadline at 4s. 2. Allow users to adjust their re-org cutoff depending on observed network conditions and their risk profile. The static 2 second cutoff was too rigid. 3. Add a `--proposer-reorg-disallowed-offsets` flag which can be used to prohibit reorgs at certain slots. This is intended to help workaround an issue whereby reorging blocks at slot 1 are currently taking ~1.6s to propagate on gossip rather than ~500ms. This is suspected to be due to a cache miss in current versions of Prysm, which should be fixed in their next release. ## Additional Info I'm of two minds about removing the `shuffling_stable` check which checks for blocks at slot 0 in the epoch. If we removed it users would be able to configure Lighthouse to try reorging at slot 0, which likely wouldn't work very well due to interactions with the proposer index cache. I think we could leave it for now and revisit it later. |
||
|
Diva M
|
911a63559b | Merge branch 'eip4844' into deneb-free-blobs | ||
|
Mac L
|
8630ddfec4 |
Add beacon.watch (#3362)
> This is currently a WIP and all features are subject to alteration or removal at any time. ## Overview The successor to #2873. Contains the backbone of `beacon.watch` including syncing code, the initial API, and several core database tables. See `watch/README.md` for more information, requirements and usage. |
||
|
realbigsean
|
fb7d729d92 | migrate types to API crate | ||
|
Michael Sproul
|
01556f6f01 |
Optimise payload attributes calculation and add SSE (#4027)
## Issue Addressed Closes #3896 Closes #3998 Closes #3700 ## Proposed Changes - Optimise the calculation of withdrawals for payload attributes by avoiding state clones, avoiding unnecessary state advances and reading from the snapshot cache if possible. - Use the execution layer's payload attributes cache to avoid re-calculating payload attributes. I actually implemented a new LRU cache just for withdrawals but it had the exact same key and most of the same data as the existing payload attributes cache, so I deleted it. - Add a new SSE event that fires when payloadAttributes are calculated. This is useful for block builders, a la https://github.com/ethereum/beacon-APIs/issues/244. - Add a new CLI flag `--always-prepare-payload` which forces payload attributes to be sent with every fcU regardless of connected proposers. This is intended for use by builders/relays. For maximum effect, the flags I've been using to run Lighthouse in "payload builder mode" are: ``` --always-prepare-payload \ --prepare-payload-lookahead 12000 \ --suggested-fee-recipient 0x0000000000000000000000000000000000000000 ``` The fee recipient is required so Lighthouse has something to pack in the payload attributes (it can be ignored by the builder). The lookahead causes fcU to be sent at the start of every slot rather than at 8s. As usual, fcU will also be sent after each change of head block. I think this combination is sufficient for builders to build on all viable heads. Often there will be two fcU (and two payload attributes) sent for the same slot: one sent at the start of the slot with the head from `n - 1` as the parent, and one sent after the block arrives with `n` as the parent. Example usage of the new event stream: ```bash curl -N "http://localhost:5052/eth/v1/events?topics=payload_attributes" ``` ## Additional Info - [x] Tests added by updating the proposer re-org tests. This has the benefit of testing the proposer re-org code paths with withdrawals too, confirming that the new changes don't interact poorly. - [ ] Benchmarking with `blockdreamer` on devnet-7 showed promising results but I'm yet to do a comparison to `unstable`. Co-authored-by: Michael Sproul <micsproul@gmail.com> |
||
|
Michael Sproul
|
16bdb2771b | Update another test broken by the shuffling change | ||
|
Michael Sproul
|
e48487db01 | Fix the new BLS to execution change test | ||
|
Mark Mackey
|
c922566fbc | Fixed Some Tests | ||
|
Michael Sproul
|
775d222299 |
Enable proposer boost re-orging (#2860)
## Proposed Changes With proposer boosting implemented (#2822) we have an opportunity to re-org out late blocks. This PR adds three flags to the BN to control this behaviour: * `--disable-proposer-reorgs`: turn aggressive re-orging off (it's on by default). * `--proposer-reorg-threshold N`: attempt to orphan blocks with less than N% of the committee vote. If this parameter isn't set then N defaults to 20% when the feature is enabled. * `--proposer-reorg-epochs-since-finalization N`: only attempt to re-org late blocks when the number of epochs since finalization is less than or equal to N. The default is 2 epochs, meaning re-orgs will only be attempted when the chain is finalizing optimally. For safety Lighthouse will only attempt a re-org under very specific conditions: 1. The block being proposed is 1 slot after the canonical head, and the canonical head is 1 slot after its parent. i.e. at slot `n + 1` rather than building on the block from slot `n` we build on the block from slot `n - 1`. 2. The current canonical head received less than N% of the committee vote. N should be set depending on the proposer boost fraction itself, the fraction of the network that is believed to be applying it, and the size of the largest entity that could be hoarding votes. 3. The current canonical head arrived after the attestation deadline from our perspective. This condition was only added to support suppression of forkchoiceUpdated messages, but makes intuitive sense. 4. The block is being proposed in the first 2 seconds of the slot. This gives it time to propagate and receive the proposer boost. ## Additional Info For the initial idea and background, see: https://github.com/ethereum/consensus-specs/pull/2353#issuecomment-950238004 There is also a specification for this feature here: https://github.com/ethereum/consensus-specs/pull/3034 Co-authored-by: Michael Sproul <micsproul@gmail.com> Co-authored-by: pawan <pawandhananjay@gmail.com> |
||
|
Paul Hauner
|
fa6ad1a11a |
Deduplicate block root computation (#3590)
## Issue Addressed NA ## Proposed Changes This PR removes duplicated block root computation. Computing the `SignedBeaconBlock::canonical_root` has become more expensive since the merge as we need to compute the merke root of each transaction inside an `ExecutionPayload`. Computing the root for [a mainnet block](https://beaconcha.in/slot/4704236) is taking ~10ms on my i7-8700K CPU @ 3.70GHz (no sha extensions). Given that our median seen-to-imported time for blocks is presently 300-400ms, removing a few duplicated block roots (~30ms) could represent an easy 10% improvement. When we consider that the seen-to-imported times include operations *after* the block has been placed in the early attester cache, we could expect the 30ms to be more significant WRT our seen-to-attestable times. ## Additional Info NA |
||
|
Paul Hauner
|
be4e261e74 |
Use async code when interacting with EL (#3244)
## Overview
This rather extensive PR achieves two primary goals:
1. Uses the finalized/justified checkpoints of fork choice (FC), rather than that of the head state.
2. Refactors fork choice, block production and block processing to `async` functions.
Additionally, it achieves:
- Concurrent forkchoice updates to the EL and cache pruning after a new head is selected.
- Concurrent "block packing" (attestations, etc) and execution payload retrieval during block production.
- Concurrent per-block-processing and execution payload verification during block processing.
- The `Arc`-ification of `SignedBeaconBlock` during block processing (it's never mutated, so why not?):
- I had to do this to deal with sending blocks into spawned tasks.
- Previously we were cloning the beacon block at least 2 times during each block processing, these clones are either removed or turned into cheaper `Arc` clones.
- We were also `Box`-ing and un-`Box`-ing beacon blocks as they moved throughout the networking crate. This is not a big deal, but it's nice to avoid shifting things between the stack and heap.
- Avoids cloning *all the blocks* in *every chain segment* during sync.
- It also has the potential to clean up our code where we need to pass an *owned* block around so we can send it back in the case of an error (I didn't do much of this, my PR is already big enough 😅)
- The `BeaconChain::HeadSafetyStatus` struct was removed. It was an old relic from prior merge specs.
For motivation for this change, see https://github.com/sigp/lighthouse/pull/3244#issuecomment-1160963273
## Changes to `canonical_head` and `fork_choice`
Previously, the `BeaconChain` had two separate fields:
```
canonical_head: RwLock<Snapshot>,
fork_choice: RwLock<BeaconForkChoice>
```
Now, we have grouped these values under a single struct:
```
canonical_head: CanonicalHead {
cached_head: RwLock<Arc<Snapshot>>,
fork_choice: RwLock<BeaconForkChoice>
}
```
Apart from ergonomics, the only *actual* change here is wrapping the canonical head snapshot in an `Arc`. This means that we no longer need to hold the `cached_head` (`canonical_head`, in old terms) lock when we want to pull some values from it. This was done to avoid deadlock risks by preventing functions from acquiring (and holding) the `cached_head` and `fork_choice` locks simultaneously.
## Breaking Changes
### The `state` (root) field in the `finalized_checkpoint` SSE event
Consider the scenario where epoch `n` is just finalized, but `start_slot(n)` is skipped. There are two state roots we might in the `finalized_checkpoint` SSE event:
1. The state root of the finalized block, which is `get_block(finalized_checkpoint.root).state_root`.
4. The state root at slot of `start_slot(n)`, which would be the state from (1), but "skipped forward" through any skip slots.
Previously, Lighthouse would choose (2). However, we can see that when [Teku generates that event](
|
||
|
Michael Sproul
|
8fa032c8ae |
Run fork choice before block proposal (#3168)
## Issue Addressed Upcoming spec change https://github.com/ethereum/consensus-specs/pull/2878 ## Proposed Changes 1. Run fork choice at the start of every slot, and wait for this run to complete before proposing a block. 2. As an optimisation, also run fork choice 3/4 of the way through the slot (at 9s), _dequeueing attestations for the next slot_. 3. Remove the fork choice run from the state advance timer that occurred before advancing the state. ## Additional Info ### Block Proposal Accuracy This change makes us more likely to propose on top of the correct head in the presence of re-orgs with proposer boost in play. The main scenario that this change is designed to address is described in the linked spec issue. ### Attestation Accuracy This change _also_ makes us more likely to attest to the correct head. Currently in the case of a skipped slot at `slot` we only run fork choice 9s into `slot - 1`. This means the attestations from `slot - 1` aren't taken into consideration, and any boost applied to the block from `slot - 1` is not removed (it should be). In the language of the linked spec issue, this means we are liable to attest to C, even when the majority voting weight has already caused a re-org to B. ### Why remove the call before the state advance? If we've run fork choice at the start of the slot then it has already dequeued all the attestations from the previous slot, which are the only ones eligible to influence the head in the current slot. Running fork choice again is unnecessary (unless we run it for the next slot and try to pre-empt a re-org, but I don't currently think this is a great idea). ### Performance Based on Prater testing this adds about 5-25ms of runtime to block proposal times, which are 500-1000ms on average (and spike to 5s+ sometimes due to state handling issues 😢 ). I believe this is a small enough penalty to enable it by default, with the option to disable it via the new flag `--fork-choice-before-proposal-timeout 0`. Upcoming work on block packing and state representation will also reduce block production times in general, while removing the spikes. ### Implementation Fork choice gets invoked at the start of the slot via the `per_slot_task` function called from the slot timer. It then uses a condition variable to signal to block production that fork choice has been updated. This is a bit funky, but it seems to work. One downside of the timer-based approach is that it doesn't happen automatically in most of the tests. The test added by this PR has to trigger the run manually. |
||
|
Age Manning
|
0aee7ec873 |
Refactor Peerdb and PeerManager (#2660)
## Proposed Changes This is a refactor of the PeerDB and PeerManager. A number of bugs have been surfacing around the connection state of peers and their interaction with the score state. This refactor tightens the mutability properties of peers such that only specific modules are able to modify the state of peer information preventing inadvertant state changes that can lead to our local peer manager db being out of sync with libp2p. Further, the logic around connection and scoring was quite convoluted and the distinction between the PeerManager and Peerdb was not well defined. Although these issues are not fully resolved, this PR is step to cleaning up this logic. The peerdb solely manages most mutability operations of peers leaving high-order logic to the peer manager. A single `update_connection_state()` function has been added to the peer-db making it solely responsible for modifying the peer's connection state. The way the peer's scores can be modified have been reduced to three simple functions (`update_scores()`, `update_gossipsub_scores()` and `report_peer()`). This prevents any add-hoc modifications of scores and only natural processes of score modification is allowed which simplifies the reasoning of score and state changes. |
||
|
Kane Wallmann
|
28b79084cd |
Fix chain_id value in config/deposit_contract RPC method (#2659)
## Issue Addressed This PR addresses issue #2657 ## Proposed Changes Changes `/eth/v1/config/deposit_contract` endpoint to return the chain ID from the loaded chain spec instead of eth1::DEFAULT_NETWORK_ID which is the Goerli chain ID of 5. Co-authored-by: Michael Sproul <michael@sigmaprime.io> |