Lion - dapplion
95cec45c38
Use data column batch verification consistently ( #6851 )
...
Resolve a `TODO(das)` to use KZG batch verification in `put_rpc_custody_columns`
Uses `verify_kzg_for_data_column_list_with_scoring` in all paths that send more than one column. To use batch verification and have attributability of which peer is sending a bad column.
Needs to move `verify_kzg_for_data_column_list_with_scoring` into the type's module to convert to the KZG verified type.
2025-02-03 06:07:45 +00:00
Lion - dapplion
027bb973f8
Compute columns in post-PeerDAS checkpoint sync ( #6760 )
...
Addresses #6026 .
Post-PeerDAS the DB expects to have data columns for the finalized block.
Instead of forcing the user to submit the columns, this PR computes the columns from the blobs that we can already fetch from the checkpointz server or with the existing CLI options.
Note 1: (EDIT) Pruning concern addressed
Note 2: I have not tested this feature
Note 3: @michaelsproul an alternative I recall is to not require the blobs / columns at this point and expect backfill to populate the finalized block
2025-01-31 06:00:52 +00:00
Eitan Seri-Levi
276eda3dfe
POST /eth/v2/beacon/pool/attestations bugfixes (#6867 )
2025-01-31 00:20:44 +00:00
Jimmy Chen
70194dfc6a
Implement PeerDAS Fulu fork activation ( #6795 )
...
Addresses #6706
This PR activates PeerDAS at the Fulu fork epoch instead of `EIP_7594_FORK_EPOCH`. This means we no longer support testing PeerDAS with Deneb / Electrs, as it's now part of a hard fork.
2025-01-30 07:01:34 +00:00
Lion - dapplion
c6ebaba892
Detect invalid proposer signature on RPC block processing ( #6519 )
...
Complements
- https://github.com/sigp/lighthouse/pull/6321
by detecting if the proposer signature is valid or not during RPC block processing. In lookup sync, if the invalid signature signature is the proposer signature, it's not deterministic on the block root. So we should only penalize the sending peer and retry. Otherwise, if it's on the body we should drop the lookup and penalize all peers that claim to have imported the block
2025-01-28 19:01:26 +00:00
Eitan Seri-Levi
a1b7d616b4
Modularize beacon node backend ( #4718 )
...
#4669
Modularize the beacon node backend to make it easier to add new database implementations
2025-01-23 02:12:16 +00:00
Jimmy Chen
f008b84079
Avoid computing columns from EL blobs if block has already been imported ( #6816 )
...
* Avoid computing columns from EL blobs if block has already been imported.
* Downgrade a `warn` log to `debug` and update handling.
2025-01-22 04:05:32 +00:00
Eitan Seri-Levi
06329ec2d1
SingleAttestation implementation (#6488 )
...
* First pass
* Add restrictions to RuntimeVariableList api
* Use empty_uninitialized and fix warnings
* Fix some todos
* Merge branch 'unstable' into max-blobs-preset
* Fix take impl on RuntimeFixedList
* cleanup
* Fix test compilations
* Fix some more tests
* Fix test from unstable
* Merge branch 'unstable' into max-blobs-preset
* SingleAttestation
* Add post attestation v2 endpoint logic to attestation service
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation
* Implement "Bugfix and more withdrawal tests"
* Implement "Add missed exit checks to consolidation processing"
* Implement "Update initial earliest_exit_epoch calculation"
* Implement "Limit consolidating balance by validator.effective_balance"
* Implement "Use 16-bit random value in validator filter"
* Implement "Do not change creds type on consolidation"
* some tests and fixed attestqtion calc
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation
* Rename PendingPartialWithdraw index field to validator_index
* Skip slots to get test to pass and add TODO
* Implement "Synchronously check all transactions to have non-zero length"
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Remove footgun function
* Minor simplifications
* Move from preset to config
* Fix typo
* Revert "Remove footgun function"
This reverts commit de01f923c7https://github.com/sigp/lighthouse into single_attestation
* Added more test coverage, simplified Attestation conversion, and other minor refactors
* Removed unusued codepaths
* Fix failing test
* Implement "Remove post-altair `initialize_beacon_state_from_eth1` from specs"
* Update preset YAML
* Remove empty RuntimeVarList awefullness
* Make max_blobs_per_block a config parameter (#6329 )
Squashed commit of the following:
commit 04b3743ec1michael@sigmaprime.io >
Date: Mon Jan 6 17:36:58 2025 +1100
Add test
commit 440e854199michael@sigmaprime.io >
Date: Mon Jan 6 17:24:50 2025 +1100
Move RuntimeFixedVector into module and rename
commit f66e179a40michael@sigmaprime.io >
Date: Mon Jan 6 17:17:17 2025 +1100
Fix release tests
commit e4bfe71cd1michael@sigmaprime.io >
Date: Mon Jan 6 17:05:30 2025 +1100
Thread through ChainSpec
commit 063b79c16amichael@sigmaprime.io >
Date: Mon Jan 6 15:32:16 2025 +1100
Try fixing tests
commit 88bedf09bcmichael@sigmaprime.io >
Date: Mon Jan 6 15:04:37 2025 +1100
Revert "Remove footgun function"
This reverts commit de01f923c732483d385bmichael@sigmaprime.io >
Date: Mon Jan 6 15:04:32 2025 +1100
Fix typo
commit 2e86585b47michael@sigmaprime.io >
Date: Mon Jan 6 15:04:15 2025 +1100
Move from preset to config
commit 1095d60a40michael@sigmaprime.io >
Date: Mon Jan 6 14:38:40 2025 +1100
Minor simplifications
commit de01f923c7michael@sigmaprime.io >
Date: Mon Jan 6 14:06:57 2025 +1100
Remove footgun function
commit 0c2c8c422421ecb58fff51a292f7michael@sigmaprime.io >
Date: Mon Jan 6 14:02:50 2025 +1100
Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
commit f51a292f77107140945+dknopik@users.noreply.github.com >
Date: Fri Jan 3 20:27:21 2025 +0100
fully lint only explicitly to avoid unnecessary rebuilds (#6753 )
* fully lint only explicitly to avoid unnecessary rebuilds
commit 7e0cddef32sora.akatsuki@gmail.com >
Date: Tue Dec 24 10:38:56 2024 +0900
Make sure we have fanout peers when publish (#6738 )
* Ensure that `fanout_peers` is always non-empty if it's `Some`
commit 21ecb58ff82fcb2935e9aefb5539pawandhananjay@gmail.com >
Date: Mon Oct 21 14:46:00 2024 -0700
Merge branch 'unstable' into max-blobs-preset
commit 2fcb2935ecpawandhananjay@gmail.com >
Date: Fri Sep 6 18:28:31 2024 -0700
Fix test from unstable
commit 12c6ef118apawandhananjay@gmail.com >
Date: Wed Sep 4 16:16:36 2024 -0700
Fix some more tests
commit d37733b846pawandhananjay@gmail.com >
Date: Wed Sep 4 12:47:36 2024 -0700
Fix test compilations
commit 52bb581e07pawandhananjay@gmail.com >
Date: Tue Sep 3 18:38:19 2024 -0700
cleanup
commit e71020e3e6pawandhananjay@gmail.com >
Date: Tue Sep 3 17:16:10 2024 -0700
Fix take impl on RuntimeFixedList
commit 13f9bba64760100fc6b4e675cf5dpawandhananjay@gmail.com >
Date: Tue Sep 3 16:08:59 2024 -0700
Merge branch 'unstable' into max-blobs-preset
commit 60100fc6bepawandhananjay@gmail.com >
Date: Fri Aug 30 16:04:11 2024 -0700
Fix some todos
commit a9cb329a22pawandhananjay@gmail.com >
Date: Fri Aug 30 15:54:00 2024 -0700
Use empty_uninitialized and fix warnings
commit 4dc6e6515epawandhananjay@gmail.com >
Date: Fri Aug 30 15:53:18 2024 -0700
Add restrictions to RuntimeVariableList api
commit 25feedfde3pawandhananjay@gmail.com >
Date: Thu Aug 29 16:11:19 2024 -0700
First pass
* Fix tests
* Implement max_blobs_per_block_electra
* Fix config issues
* Simplify BlobSidecarListFromRoot
* Disable PeerDAS tests
* Cleanup single attestation imports
* Fix some single attestation network plumbing
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Bump quota to account for new target (6)
* Remove clone
* Fix issue from review
* Try to remove ugliness
* Merge branch 'unstable' into max-blobs-preset
* Merge remote-tracking branch 'origin/unstable' into electra-alpha10
* Merge commit '04b3743ec1e0b650269dd8e58b540c02430d1c0d' into electra-alpha10
* Merge remote-tracking branch 'pawan/max-blobs-preset' into electra-alpha10
* Update tests to v1.5.0-beta.0
* Merge remote-tracking branch 'origin/electra-alpha10' into single_attestation
* Fix some tests
* Cargo fmt
* lint
* fmt
* Resolve merge conflicts
* Merge branch 'electra-alpha10' of https://github.com/sigp/lighthouse into single_attestation
* lint
* Linting
* fmt
* Merge branch 'electra-alpha10' of https://github.com/sigp/lighthouse into single_attestation
* Fmt
* Fix test and add TODO
* Gracefully handle slashed proposers in fork choice tests
* Merge remote-tracking branch 'origin/unstable' into electra-alpha10
* Keep latest changes from max_blobs_per_block PR in codec.rs
* Revert a few more regressions and add a comment
* Merge branch 'electra-alpha10' of https://github.com/sigp/lighthouse into single_attestation
* Disable more DAS tests
* Improve validator monitor test a little
* Make test more robust
* Fix sync test that didn't understand blobs
* Fill out cropped comment
* Merge remote-tracking branch 'origin/electra-alpha10' into single_attestation
* Merge remote-tracking branch 'origin/unstable' into single_attestation
* Merge remote-tracking branch 'origin/unstable' into single_attestation
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into single_attestation
* publish_attestations should accept Either<Attestation,SingleAttestation>
* log an error when failing to convert to SingleAttestation
* Use Cow to avoid clone
* Avoid reconverting to SingleAttestation
* Tweak VC error message
* update comments
* update comments
* pass in single attestation as ref to subnetid calculation method
* Improved API, new error variants and other minor tweaks
* Fix single_attestation event topic boilerplate
* fix sse event failure
* Add single_attestation event topic test coverage
2025-01-16 18:27:08 +00:00
Jimmy Chen
e98209d118
Implement PeerDAS subnet decoupling (aka custody groups) ( #6736 )
...
* Implement PeerDAS subnet decoupling (aka custody groups).
* Merge branch 'unstable' into decouple-subnets
* Refactor feature testing for spec tests (#6737 )
Squashed commit of the following:
commit 898d05ee17ffbd25e2b7e0cddef3jchen.tc@gmail.com >
Date: Tue Dec 24 14:41:19 2024 +1100
Merge branch 'unstable' into refactor-ef-tests-features
commit ffbd25e2bejchen.tc@gmail.com >
Date: Tue Dec 24 14:40:38 2024 +1100
Fix `SszStatic` tests for PeerDAS: exclude eip7594 test vectors when testing Electra types.
commit aa593cf35cjchen.tc@gmail.com >
Date: Fri Dec 20 12:08:54 2024 +1100
Refactor spec testing for features and simplify usage.
* Fix build.
* Add input validation and improve arithmetic handling when calculating custody groups.
* Address review comments re code style consistency.
* Merge branch 'unstable' into decouple-subnets
# Conflicts:
# beacon_node/beacon_chain/src/kzg_utils.rs
# beacon_node/beacon_chain/src/observed_data_sidecars.rs
# beacon_node/lighthouse_network/src/discovery/subnet_predicate.rs
# common/eth2_network_config/built_in_network_configs/chiado/config.yaml
# common/eth2_network_config/built_in_network_configs/gnosis/config.yaml
# common/eth2_network_config/built_in_network_configs/holesky/config.yaml
# common/eth2_network_config/built_in_network_configs/mainnet/config.yaml
# common/eth2_network_config/built_in_network_configs/sepolia/config.yaml
# consensus/types/src/chain_spec.rs
* Update consensus/types/src/chain_spec.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com >
* Merge remote-tracking branch 'origin/unstable' into decouple-subnets
* Update error handling.
* Address review comment.
* Merge remote-tracking branch 'origin/unstable' into decouple-subnets
# Conflicts:
# consensus/types/src/chain_spec.rs
* Update PeerDAS spec tests to `1.5.0-beta.0` and fix failing unit tests.
* Merge remote-tracking branch 'origin/unstable' into decouple-subnets
# Conflicts:
# beacon_node/lighthouse_network/src/peer_manager/mod.rs
2025-01-15 07:40:26 +00:00
Jimmy Chen
dd7591f712
Fix data columns not persisting for PeerDAS due to a getBlobs race condition ( #6756 )
...
* Fix data columns not persisting for PeerDAS due to a `getBlobs` race condition.
* Refactor blobs and columns logic in `chain.import_block` for clarity. Add more docs on `data_column_recv`.
* Add more code comments for clarity.
* Merge remote-tracking branch 'origin/unstable' into fix-column-race
# Conflicts:
# beacon_node/beacon_chain/src/block_verification_types.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
* Fix lint.
2025-01-15 06:56:51 +00:00
Michael Sproul
06e4d22d49
Electra spec changes for v1.5.0-beta.0 ( #6731 )
...
* First pass
* Add restrictions to RuntimeVariableList api
* Use empty_uninitialized and fix warnings
* Fix some todos
* Merge branch 'unstable' into max-blobs-preset
* Fix take impl on RuntimeFixedList
* cleanup
* Fix test compilations
* Fix some more tests
* Fix test from unstable
* Merge branch 'unstable' into max-blobs-preset
* Implement "Bugfix and more withdrawal tests"
* Implement "Add missed exit checks to consolidation processing"
* Implement "Update initial earliest_exit_epoch calculation"
* Implement "Limit consolidating balance by validator.effective_balance"
* Implement "Use 16-bit random value in validator filter"
* Implement "Do not change creds type on consolidation"
* Rename PendingPartialWithdraw index field to validator_index
* Skip slots to get test to pass and add TODO
* Implement "Synchronously check all transactions to have non-zero length"
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Remove footgun function
* Minor simplifications
* Move from preset to config
* Fix typo
* Revert "Remove footgun function"
This reverts commit de01f923c7#6329 )
Squashed commit of the following:
commit 04b3743ec1michael@sigmaprime.io >
Date: Mon Jan 6 17:36:58 2025 +1100
Add test
commit 440e854199michael@sigmaprime.io >
Date: Mon Jan 6 17:24:50 2025 +1100
Move RuntimeFixedVector into module and rename
commit f66e179a40michael@sigmaprime.io >
Date: Mon Jan 6 17:17:17 2025 +1100
Fix release tests
commit e4bfe71cd1michael@sigmaprime.io >
Date: Mon Jan 6 17:05:30 2025 +1100
Thread through ChainSpec
commit 063b79c16amichael@sigmaprime.io >
Date: Mon Jan 6 15:32:16 2025 +1100
Try fixing tests
commit 88bedf09bcmichael@sigmaprime.io >
Date: Mon Jan 6 15:04:37 2025 +1100
Revert "Remove footgun function"
This reverts commit de01f923c732483d385bmichael@sigmaprime.io >
Date: Mon Jan 6 15:04:32 2025 +1100
Fix typo
commit 2e86585b47michael@sigmaprime.io >
Date: Mon Jan 6 15:04:15 2025 +1100
Move from preset to config
commit 1095d60a40michael@sigmaprime.io >
Date: Mon Jan 6 14:38:40 2025 +1100
Minor simplifications
commit de01f923c7michael@sigmaprime.io >
Date: Mon Jan 6 14:06:57 2025 +1100
Remove footgun function
commit 0c2c8c422421ecb58fff51a292f7michael@sigmaprime.io >
Date: Mon Jan 6 14:02:50 2025 +1100
Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
commit f51a292f77107140945+dknopik@users.noreply.github.com >
Date: Fri Jan 3 20:27:21 2025 +0100
fully lint only explicitly to avoid unnecessary rebuilds (#6753 )
* fully lint only explicitly to avoid unnecessary rebuilds
commit 7e0cddef32sora.akatsuki@gmail.com >
Date: Tue Dec 24 10:38:56 2024 +0900
Make sure we have fanout peers when publish (#6738 )
* Ensure that `fanout_peers` is always non-empty if it's `Some`
commit 21ecb58ff82fcb2935e9aefb5539pawandhananjay@gmail.com >
Date: Mon Oct 21 14:46:00 2024 -0700
Merge branch 'unstable' into max-blobs-preset
commit 2fcb2935ecpawandhananjay@gmail.com >
Date: Fri Sep 6 18:28:31 2024 -0700
Fix test from unstable
commit 12c6ef118apawandhananjay@gmail.com >
Date: Wed Sep 4 16:16:36 2024 -0700
Fix some more tests
commit d37733b846pawandhananjay@gmail.com >
Date: Wed Sep 4 12:47:36 2024 -0700
Fix test compilations
commit 52bb581e07pawandhananjay@gmail.com >
Date: Tue Sep 3 18:38:19 2024 -0700
cleanup
commit e71020e3e6pawandhananjay@gmail.com >
Date: Tue Sep 3 17:16:10 2024 -0700
Fix take impl on RuntimeFixedList
commit 13f9bba64760100fc6b4e675cf5dpawandhananjay@gmail.com >
Date: Tue Sep 3 16:08:59 2024 -0700
Merge branch 'unstable' into max-blobs-preset
commit 60100fc6bepawandhananjay@gmail.com >
Date: Fri Aug 30 16:04:11 2024 -0700
Fix some todos
commit a9cb329a22pawandhananjay@gmail.com >
Date: Fri Aug 30 15:54:00 2024 -0700
Use empty_uninitialized and fix warnings
commit 4dc6e6515epawandhananjay@gmail.com >
Date: Fri Aug 30 15:53:18 2024 -0700
Add restrictions to RuntimeVariableList api
commit 25feedfde3pawandhananjay@gmail.com >
Date: Thu Aug 29 16:11:19 2024 -0700
First pass
* Fix tests
* Implement max_blobs_per_block_electra
* Fix config issues
* Simplify BlobSidecarListFromRoot
* Disable PeerDAS tests
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Bump quota to account for new target (6)
* Remove clone
* Fix issue from review
* Try to remove ugliness
* Merge branch 'unstable' into max-blobs-preset
* Merge remote-tracking branch 'origin/unstable' into electra-alpha10
* Merge commit '04b3743ec1e0b650269dd8e58b540c02430d1c0d' into electra-alpha10
* Merge remote-tracking branch 'pawan/max-blobs-preset' into electra-alpha10
* Update tests to v1.5.0-beta.0
* Resolve merge conflicts
* Linting
* fmt
* Fix test and add TODO
* Gracefully handle slashed proposers in fork choice tests
* Merge remote-tracking branch 'origin/unstable' into electra-alpha10
* Keep latest changes from max_blobs_per_block PR in codec.rs
* Revert a few more regressions and add a comment
* Disable more DAS tests
* Improve validator monitor test a little
* Make test more robust
* Fix sync test that didn't understand blobs
* Fill out cropped comment
2025-01-13 23:17:00 +00:00
Pawan Dhananjay
05727290fb
Make max_blobs_per_block a config parameter ( #6329 )
...
* First pass
* Add restrictions to RuntimeVariableList api
* Use empty_uninitialized and fix warnings
* Fix some todos
* Merge branch 'unstable' into max-blobs-preset
* Fix take impl on RuntimeFixedList
* cleanup
* Fix test compilations
* Fix some more tests
* Fix test from unstable
* Merge branch 'unstable' into max-blobs-preset
* Merge remote-tracking branch 'origin/unstable' into max-blobs-preset
* Remove footgun function
* Minor simplifications
* Move from preset to config
* Fix typo
* Revert "Remove footgun function"
This reverts commit de01f923c7
2025-01-10 06:34:58 +00:00
Mac L
ecdf2d891f
Add Fulu boilerplate ( #6695 )
...
* Add Fulu boilerplate
* Add more boilerplate
* Change fulu_time to osaka_time
* Merge branch 'unstable' into fulu-boilerplate
* Fix tests
* Merge branch 'unstable' into fulu-boilerplate
* More test fixes
* Apply suggestions
* Remove `get_payload` boilerplate
* Add lightclient fulu types and fix beacon-chain-tests
* Disable Fulu in ef-tests
* Reduce boilerplate for future forks
* Small fixes
* One more fix
* Apply suggestions
* Merge branch 'unstable' into fulu-boilerplate
* Fix lints
2025-01-10 05:25:23 +00:00
Pawan Dhananjay
1f6850fae2
Rust 1.84 lints ( #6781 )
...
* Fix few lints
* Fix remaining lints
* Use fully qualified syntax
2025-01-10 01:13:29 +00:00
Jimmy Chen
80cfbea7fe
Fix incorrect data column metric name ( #6761 )
...
* Fix incorrect data column metric name.
2025-01-08 03:12:37 +00:00
Jimmy Chen
7ec748a108
Implement getBlobSidecars support for PeerDAS ( #6755 )
...
* Implement getBlobSidecars endpoint for PeerDAS.
* Merge branch 'unstable' into peerdas-get-blob-sidecars
* Fix incorrect logging.
* Replace `and_then` usage.
2025-01-08 03:12:34 +00:00
Ekaterina Riazantseva
57141d8b4b
Add 'beacon_' prefix to PeerDAS metrics names ( #6537 )
...
* Add 'beacon_' prefix to PeerDAS metrics names
* Merge remote-tracking branch 'origin/unstable' into peerdas-metrics
* Merge 'origin/unstable' into peerdas-metrics
* Merge remote-tracking branch 'origin/unstable/ into peerdas-metrics
* Add 'beacon_' prefix to 'kzg_data_column' metrics
2025-01-08 00:39:53 +00:00
Michael Sproul
07e82dabc0
Delete OTB verification service ( #6631 )
...
* Delete OTB verification service
* Merge branch 'unstable' into delete-otb
2024-12-19 05:46:06 +00:00
Mac L
b2b1faad4e
Enforce alphabetically ordered cargo deps ( #6678 )
...
* Enforce alphabetically ordered cargo deps
* Fix test-suite
* Another CI fix
* Merge branch 'unstable' into cargo-sort
* Fix conflicts
* Merge remote-tracking branch 'origin/unstable' into cargo-sort
2024-12-19 05:46:03 +00:00
Pawan Dhananjay
2662dc7f8f
Fix Sse client api ( #6685 )
...
* Use reqwest eventsource for get_events api
* await for Event::Open before returning stream
* fmt
* Merge branch 'unstable' into sse-client-fix
* Ignore lint
2024-12-18 00:05:58 +00:00
Daniel Knopik
02cb2d68ff
Enable lints for tests only running optimized ( #6664 )
...
* enable linting optimized-only tests
* fix automatically fixable or obvious lints
* fix suspicious_open_options by removing manual options
* fix `await_holding_lock`s
* avoid failing lint due to now disabled `#[cfg(debug_assertions)]`
* reduce future sizes in tests
* fix accidently flipped assert logic
* restore holding lock for web3signer download
* Merge branch 'unstable' into lint-opt-tests
2024-12-17 00:40:35 +00:00
ethDreamer
86891e6d0f
builder gas limit & some refactoring ( #6583 )
...
* Cache gas_limit
* Payload Parameters Refactor
* Enforce Proposer Gas Limit
* Fixed and Added New Tests
* Fix Beacon Chain Tests
2024-12-16 05:43:58 +00:00
Michael Sproul
a6de0d5e12
Merge remote-tracking branch 'origin/stable' into back-merge-6.0.1
2024-12-16 15:37:52 +11:00
Lion - dapplion
c92c07ff49
Track beacon processor import result metrics ( #6541 )
...
* Track beacon processor import result metrics
* Update metric name
2024-12-16 04:33:33 +00:00
Michael Sproul
f3b78889e5
Compact more when pruning states ( #6667 )
...
* Compact more when pruning states
* Merge branch 'release-v6.0.1' into compact-more
2024-12-14 08:43:00 +00:00
Michael Sproul
fc0e0ae613
Prevent reconstruction starting prematurely ( #6669 )
...
* Prevent reconstruction starting prematurely
* Simplify condition
* Merge remote-tracking branch 'origin/release-v6.0.1' into dont-start-reconstruction-early
2024-12-12 01:58:41 +00:00
Michael Sproul
b2590bcb37
Tweak reconstruction batch size ( #6668 )
...
* Tweak reconstruction batch size
* Merge branch 'release-v6.0.1' into reconstruction-batch-size
2024-12-11 22:51:46 +00:00
Michael Sproul
fa6c4c02a3
Fix Rust 1.83 Clippy lints ( #6629 )
...
* Fix Rust 1.83 Clippy lints
* Cargo fmt
2024-11-29 02:23:54 +00:00
Michael Sproul
b1e9f69460
Fix v22 schema upgrade ( #6591 )
...
* Fix v22 schema upgrade
* Ownership
2024-11-19 22:43:18 +00:00
Michael Sproul
9fdd53df56
Hierarchical state diffs ( #5978 )
...
* Start extracting freezer changes for tree-states
* Remove unused config args
* Add comments
* Remove unwraps
* Subjective more clear implementation
* Clean up hdiff
* Update xdelta3
* Tree states archive metrics (#6040 )
* Add store cache size metrics
* Add compress timer metrics
* Add diff apply compute timer metrics
* Add diff buffer cache hit metrics
* Add hdiff buffer load times
* Add blocks replayed metric
* Move metrics to store
* Future proof some metrics
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io >
* Port and clean up forwards iterator changes
* Add and polish hierarchy-config flag
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Cleaner errors
* Fix beacon_chain test compilation
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Patch a few more freezer block roots
* Fix genesis block root bug
* Fix test failing due to pending updates
* Beacon chain tests passing
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix doc lint
* Implement DB schema upgrade for hierarchical state diffs (#6193 )
* DB upgrade
* Add flag
* Delete RestorePointHash
* Update docs
* Update docs
* Implement hierarchical state diffs config migration (#6245 )
* Implement hierarchical state diffs config migration
* Review PR
* Remove TODO
* Set CURRENT_SCHEMA_VERSION correctly
* Fix genesis state loading
* Re-delete some PartialBeaconState stuff
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io >
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix test compilation
* Update schema downgrade test
* Fix tests
* Fix null anchor migration
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix tree states upgrade migration (#6328 )
* Towards crash safety
* Fix compilation
* Move cold summaries and state roots to new columns
* Rename StateRoots chunked field
* Update prune states
* Clean hdiff CLI flag and metrics
* Fix "staged reconstruction"
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Fix alloy issues
* Fix staged reconstruction logic
* Prevent weird slot drift
* Remove "allow" flag
* Update CLI help
* Remove FIXME about downgrade
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Remove some unnecessary error variants
* Fix new test
* Tree states archive - review comments and metrics (#6386 )
* Review PR comments and metrics
* Comments
* Add anchor metrics
* drop prev comment
* Update metadata.rs
* Apply suggestions from code review
---------
Co-authored-by: Michael Sproul <micsproul@gmail.com >
* Update beacon_node/store/src/hot_cold_store.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com >
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Clarify comment and remove anchor_slot garbage
* Simplify database anchor (#6397 )
* Simplify database anchor
* Update beacon_node/store/src/reconstruct.rs
* Add migration for anchor
* Fix and simplify light_client store tests
* Fix incompatible config test
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* More metrics
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* New historic state cache (#6475 )
* New historic state cache
* Add more metrics
* State cache hit rate metrics
* Fix store metrics
* More logs and metrics
* Fix logger
* Ensure cached states have built caches :O
* Replay blocks in preference to diffing
* Two separate caches
* Distribute cache build time to next slot
* Re-plumb historic-state-cache flag
* Clean up metrics
* Update book
* Update beacon_node/store/src/hdiff.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com >
* Update beacon_node/store/src/historic_state_cache.rs
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com >
---------
Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com >
* Update database docs
* Update diagram
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Update lockbud to work with bindgen/etc
* Correct pkg name for Debian
* Remove vestigial epochs_per_state_diff
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Markdown lint
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Address Jimmy's review comments
* Simplify ReplayFrom case
* Fix and document genesis_state_root
* Typo
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com >
* Merge branch 'unstable' into tree-states-archive
* Compute diff of validators list manually (#6556 )
* Split hdiff computation
* Dedicated logic for historical roots and summaries
* Benchmark against real states
* Mutated source?
* Version the hdiff
* Add lighthouse DB config for hierarchy exponents
* Tidy up hierarchy exponents flag
* Apply suggestions from code review
Co-authored-by: Michael Sproul <micsproul@gmail.com >
* Address PR review
* Remove hardcoded paths in benchmarks
* Delete unused function in benches
* lint
---------
Co-authored-by: Michael Sproul <michael@sigmaprime.io >
* Test hdiff binary format stability (#6585 )
* Merge remote-tracking branch 'origin/unstable' into tree-states-archive
* Add deprecation warning for SPRP
* Update xdelta to get rid of duplicate deps
* Document test
2024-11-18 01:51:44 +00:00
Eitan Seri-Levi
654fc6acdc
Additional light client metrics ( #6545 )
...
* Fix db query and add some additional metrics
* fmt
* Update beacon_node/beacon_chain/src/metrics.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com >
* Update beacon_node/beacon_chain/src/metrics.rs
Co-authored-by: Jimmy Chen <jchen.tc@gmail.com >
2024-11-15 07:09:54 +00:00
Jimmy Chen
5f053b0b6d
Improving blob propagation post-PeerDAS with Decentralized Blob Building ( #6268 )
...
* Get blobs from EL.
Co-authored-by: Michael Sproul <michael@sigmaprime.io >
* Avoid cloning blobs after fetching blobs.
* Address review comments and refactor code.
* Fix lint.
* Move blob computation metric to the right spot.
* Merge branch 'unstable' into das-fetch-blobs
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
# beacon_node/beacon_chain/src/block_verification.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
* Gradual publication of data columns for supernodes.
* Recompute head after importing block with blobs from the EL.
* Fix lint
* Merge branch 'unstable' into das-fetch-blobs
* Use blocking task instead of async when computing cells.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs
* Fix semantic conflicts
* Downgrade error log.
* Merge branch 'unstable' into das-fetch-blobs
# Conflicts:
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/execution_layer/src/engine_api.rs
# beacon_node/execution_layer/src/engine_api/json_structures.rs
# beacon_node/network/src/network_beacon_processor/gossip_methods.rs
# beacon_node/network/src/network_beacon_processor/mod.rs
# beacon_node/network/src/network_beacon_processor/sync_methods.rs
* Merge branch 'unstable' into das-fetch-blobs
* Publish block without waiting for blob and column proof computation.
* Address review comments and refactor.
* Merge branch 'unstable' into das-fetch-blobs
* Fix test and docs.
* Comment cleanups.
* Merge branch 'unstable' into das-fetch-blobs
* Address review comments and cleanup
* Address review comments and cleanup
* Refactor to de-duplicate gradual publication logic.
* Add more logging.
* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs
# Conflicts:
# Cargo.lock
* Fix incorrect comparison on `num_fetched_blobs`.
* Implement gradual blob publication.
* Merge branch 'unstable' into das-fetch-blobs
* Inline `publish_fn`.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
* Gossip verify blobs before publishing
* Avoid queries for 0 blobs and error for duplicates
* Gossip verified engine blob before processing them, and use observe cache to detect duplicates before publishing.
* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs
# Conflicts:
# beacon_node/network/src/network_beacon_processor/mod.rs
* Merge branch 'unstable' into das-fetch-blobs
* Fix invalid commitment inclusion proofs in blob sidecars created from EL blobs.
* Only publish EL blobs triggered from gossip block, and not RPC block.
* Downgrade gossip blob log to `debug`.
* Merge branch 'unstable' into das-fetch-blobs
* Merge branch 'unstable' into das-fetch-blobs
* Grammar
2024-11-15 03:34:13 +00:00
Lion - dapplion
38388979db
Strict match of errors in backfill sync ( #6520 )
...
* Strict match of errors in backfill sync
* Fix tests
2024-11-05 01:00:10 +00:00
Michael Sproul
fe889c619c
Simplify light client tests ( #6542 )
...
* Simplify light client tests
2024-10-29 01:31:36 +00:00
Age Manning
e31ac508d4
Modularize tracing executor and metrics rename ( #6424 )
...
* Tracing executor and metrics rename
* Appease clippy
* Merge branch 'unstable' into modularise-task-executor
2024-10-28 09:41:45 +00:00
Eitan Seri-Levi
9d069a9588
Fix electra light client types ( #6361 )
...
* persist light client updates
* update beacon chain to serve light client updates
* resolve todos
* cache best update
* extend cache parts
* is better light client update
* resolve merge conflict
* initial api changes
* add lc update db column
* fmt
* added tests
* add sim
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* fix some weird issues with the simulator
* tests
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* test changes
* merge conflict
* testing
* started work on ef tests and some code clean up
* update tests
* linting
* noop pre altair, were still failing on electra though
* allow for zeroed light client header
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-updates
* merge unstable
* remove unwraps
* remove unwraps
* fetch bootstrap without always querying for state
* storing bootstrap parts in db
* mroe code cleanup
* test
* prune sync committee branches from dropped chains
* Update light_client_update.rs
* merge unstable
* move functionality to helper methods
* refactor is best update fn
* refactor is best update fn
* improve organization of light client server cache logic
* fork diget calc, and only spawn as many blcoks as we need for the lc update test
* resovle merge conflict
* add electra bootstrap logic, add logic to cache current sync committee
* add latest sync committe branch cache
* fetch lc update from the cache if it exists
* fmt
* Fix beacon_chain tests
* Add debug code to update ranking_order ef test
* Fix compare code
* merge conflicts
* merge conflict
* add better error messaging
* resolve merge conflicts
* remove lc update from basicsim
* rename sync comittte variable and fix persist condition
* refactor get_light_client_update logic
* add better comments, return helpful error messages over http and rpc
* pruning canonical non checkpoint slots
* fix test
* rerun test
* update pruning logic, add tests
* fix tests
* fix imports
* fmt
* refactor db code
* Refactor db method
* Refactor db method
* lc electra changes
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light-client-electra
* add additional comments
* testing lc merkle changes
* lc electra
* update struct defs
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light-client-electra
* fix merge
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into persist-light-client-bootstrap
* fix merge
* linting
* merge conflict
* prevent overflow
* enable lc server for http api tests
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light-client-electra
* get tests working:
* remove related TODOs
* fix test lint
* Merge branch 'persist-light-client-bootstrap' of https://github.com/eserilev/lighthouse into light-client-electra
* fix tests
* fix conflicts
* remove prints
* Merge branch 'persist-light-client-bootstrap' of https://github.com/eserilev/lighthouse into light-client-electra
* remove warning
* resolve conflicts
* merge conflicts
* linting
* remove comments
* cleanup
* linting
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light-client-electra
* pre/post electra light client cached data
* add proof type alias
* move is_empty_branch method out of impl
* add ssz tests for all forks
* refactor beacon state proof codepaths
* rename method
* fmt
* clean up proof logic
* refactor merkle proof api
* fmt
* Merge branch 'unstable' into light-client-electra
* Use superstruct mapping macros
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light-client-electra
* rename proof to merkleproof
* fmt
* Resolve merge conflicts
* merge conflicts
2024-10-25 05:19:13 +00:00
Michael Sproul
a732a87846
Remove TTD flags and safe-slots-to-import-* ( #6489 )
...
* Delete SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY
* Update fork choice tests
* Remove TTD related flags
* Add deprecation warning
* Remove more dead code
* Delete EF on_merge_block tests
* Remove even more dead code
* Address Mac's review comments
2024-10-21 01:28:55 +00:00
Jimmy Chen
ee7fca3ebd
Refactor data column reconstruction and avoid blocking processing ( #6403 )
...
* Move reconstruction logic out of `overflow_lru_cache` to simplify the code and avoids having to pass `DataColumnsToPublish` around and blocking other processing.
* Publish reconstructed cells before recomputing head. Remove duplicate functions.
* Merge branch 'unstable' into non-blocking-reconstruction
* Merge branch 'unstable' into non-blocking-reconstruction
# Conflicts:
# beacon_node/beacon_chain/src/beacon_chain.rs
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
# beacon_node/network/src/network_beacon_processor/sync_methods.rs
* Spawn a blocking task for reconstruction.
* Merge branch 'unstable' into non-blocking-reconstruction
# Conflicts:
# beacon_node/network/src/network_beacon_processor/mod.rs
* Fix fmt
* Merge branch 'unstable' into non-blocking-reconstruction
# Conflicts:
# beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
* Fix race condition by making check and mutation atomic as suggested by Lion. Also added error handling to reconstruction failure.
* Add reconstruction reason metric and more debug logging to da checker.
* Add comment and logging.
* Rename `NotRequired` to `NotStarted`.
* Remove extra character added.
2024-10-17 04:56:25 +00:00
Pawan Dhananjay
83d5c521d7
Electra updates for v1.5.0-alpha.6 ( #6445 )
...
* Update process_slashing
* Update test vectors version
* Delete Domain::Consolidation
* Rename to get_max_effective_balance
* Fix unused; lint
* Add the pre-electra slashing processing
* lint
* Change request json types
* Move requests from payload to beacon block body
* Refactor engine api
* Fix warnings
* Update engine api to latest
* engine api changed..again
* yet again
* Merge branch 'engine-requests' into electra-updates
* Fix tests
* Store reference instead of bytes in NewPayloadRequest
* Merge branch 'unstable' into electra-updates
* Update beacon_node/execution_layer/src/engine_api/json_structures.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com >
* Update beacon_node/execution_layer/src/lib.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com >
* Update beacon_node/execution_layer/src/test_utils/handle_rpc.rs
Co-authored-by: Michael Sproul <micsproul@gmail.com >
2024-10-15 17:38:43 +00:00
Lion - dapplion
71c5388461
Transition block lookup sync to range sync ( #6122 )
...
* Transition block lookup sync to range sync
* Log unexpected state
* Merge remote-tracking branch 'sigp/unstable' into lookup-to-range
* Add docs
* Merge remote-tracking branch 'sigp/unstable' into lookup-to-range
2024-10-08 21:18:41 +00:00
Michael Sproul
1bd8f31545
Clean up temporary state flags while running ( #6422 )
...
* Clean up temporary state flags while running
* Add regression test
* Simplify
2024-10-07 09:41:52 +00:00
Jimmy Chen
f3a5e256da
Implement Subnet Sampling for PeerDAS ( #6410 )
...
* Add `SAMPLES_PER_SLOT` config.
* Rename `sampling` module to `peer_sampling`
* Implement subnet sampling.
* Update lookup test.
* Merge branch 'unstable' into subnet-sampling
* Merge branch 'unstable' into subnet-sampling
# Conflicts:
# beacon_node/beacon_chain/src/data_availability_checker.rs
# beacon_node/http_api/src/publish_blocks.rs
# beacon_node/lighthouse_network/src/types/globals.rs
# beacon_node/network/src/sync/manager.rs
* Merge branch 'unstable' into subnet-sampling
2024-10-04 00:27:30 +00:00
Akihito Nakano
a4a673b780
Output network-test logs into files in CI ( #6355 )
...
* Add ci_logger
* Update artifact name
* Add env var
* Add fork_name
* Fix clippy error
* Add comments
2024-10-03 13:53:36 +00:00
Jimmy Chen
17849b58ec
Fix invalid data column sidecars getting accepted ( #6454 )
...
* Fix invalid data column sidecars getting accepted.
* Update code to match spec function.
2024-10-03 03:06:05 +00:00
Eitan Seri-Levi
82faf975b3
Add {fork_name}_enabled functions ( #5951 )
...
* add fork_name_enabled fn to Forkname impl
* refactor codebase to use new fork_enabled fn
* fmt
* Merge branch 'unstable' of https://github.com/sigp/lighthouse into fork-ord-impl
* small code cleanup
* resolve merge conflicts
* fix beacon chain test
* merge conflicts
* fix ef test issue
* resolve merge conflicts
2024-10-03 02:00:52 +00:00
Michael Sproul
2792705331
Lenient duplicate checks on HTTP API for block publication ( #5574 )
...
* start splitting gossip verification
* WIP
* Gossip verify separate (#7 )
* save
* save
* make ProvenancedBlock concrete
* delete into gossip verified block contents
* get rid of IntoBlobSidecar trait
* remove IntoGossipVerified trait
* get tests compiling
* don't check sidecar slashability in publish
* remove second publish closure
* drop blob bool. also prefer using message index over index of position in list
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix low-hanging tests
* Fix tests and clean up
* Clean up imports
* more cleanup
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Further refine behaviour and add tests
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Remove empty line
* Fix test (block is not fully imported just gossip verified)
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Update for unstable & use empty blob list
* Update comment
* Add test for duplicate block case
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Clarify unreachable case
* Fix another publish_block case
* Remove unreachable case in filter chain segment
* Revert unrelated blob optimisation
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix merge conflicts
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix some compilation issues. Impl is fucked though
* Support peerDAS
* Fix tests
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Fix conflict
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
* Address review comments
* Merge remote-tracking branch 'origin/unstable' into gossip-verify-separate
2024-09-24 04:52:44 +00:00
Michael Sproul
1447eeb40b
Improve single-node testnet support and Arc NetworkConfig/ChainSpec ( #6396 )
...
* Arc ChainSpec and NetworkConfig
* Fix release tests
* Fix lint
* Merge remote-tracking branch 'origin/unstable' into single-node-testnet
2024-09-24 00:16:18 +00:00
Lion - dapplion
d84df5799c
Attribute invalid column proof error to correct peer ( #6377 )
...
* Attribute invalid column proof error to correct peer
* Update beacon_node/beacon_chain/src/data_availability_checker.rs
Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com >
* fix conflicts
2024-09-23 18:49:26 +00:00
Eitan Seri-Levi
b619f1ab5c
Make BeaconChain::kzg field mandatory ( #6267 )
...
* make kzg field required
* update todo
* always load trusted setup WIP
* fmt
* use new rust_eth_kzg version
* merge conlficts
* add kzg fn with trusted setup disabled
* as_slice
* add kzg with no precomp
* ignore udep for kzg
* refactor kzg init
* fix peerdas kzg schedule
* fix
* udeps
* uuuudeps
* merge conflict resolved
* merge conflict
* merge conflicts
* resolve TODO
* update
* move kzg to a test util fn
* remove trusted setup default impl
* lint fmt
* fix failing test
* lint
* fix test
* Merge branch 'unstable' into beacon-chain-kzg-field-required
2024-09-23 04:54:32 +00:00
Jimmy Chen
46e0d66e2d
Fix deadlock on block cache. ( #6412 )
...
* Fix deadlock on block cache.
2024-09-19 04:58:43 +00:00
