Commit Graph

3868 Commits

Author SHA1 Message Date
Jimmy Chen
80cfbea7fe Fix incorrect data column metric name (#6761)
* Fix incorrect data column metric name.
2025-01-08 03:12:37 +00:00
Jimmy Chen
7ec748a108 Implement getBlobSidecars support for PeerDAS (#6755)
* Implement getBlobSidecars endpoint for PeerDAS.

* Merge branch 'unstable' into peerdas-get-blob-sidecars

* Fix incorrect logging.

* Replace `and_then` usage.
2025-01-08 03:12:34 +00:00
Ekaterina Riazantseva
57141d8b4b Add 'beacon_' prefix to PeerDAS metrics names (#6537)
* Add 'beacon_' prefix to PeerDAS metrics names

* Merge remote-tracking branch 'origin/unstable' into peerdas-metrics

* Merge 'origin/unstable' into peerdas-metrics

* Merge remote-tracking branch 'origin/unstable/ into peerdas-metrics

* Add 'beacon_' prefix to 'kzg_data_column' metrics
2025-01-08 00:39:53 +00:00
Eitan Seri-Levi
63a26b1107 Linting 2025-01-07 13:24:29 +07:00
jacobkaufmann
d88f1021ea Merge branch 'unstable' into electra-focil 2024-12-26 12:54:09 -05:00
Akihito Nakano
7e0cddef32 Make sure we have fanout peers when publish (#6738)
* Ensure that `fanout_peers` is always non-empty if it's `Some`
2024-12-24 01:38:56 +00:00
jacobkaufmann
9e9b6b776d Merge branch 'stable' into electra-focil 2024-12-22 20:22:01 -07:00
jacobkaufmann
985382e3e5 add IL service to download IL from EL, sign in VC, and publish via BN
add HTTP API to the beacon node to retrieve IL from the EL.
add IL service in the validator client to download the IL from the beacon node.
add logic to the beacon node to package the IL for the validator client.
add HTTP API to the beacon node to gossip signed ILs.
the validator client will sign the ILs from the beacon node and resubmit to the beacon node to gossip.
2024-12-19 14:23:36 -07:00
Michael Sproul
07e82dabc0 Delete OTB verification service (#6631)
* Delete OTB verification service

* Merge branch 'unstable' into delete-otb
2024-12-19 05:46:06 +00:00
Mac L
b2b1faad4e Enforce alphabetically ordered cargo deps (#6678)
* Enforce alphabetically ordered cargo deps

* Fix test-suite

* Another CI fix

* Merge branch 'unstable' into cargo-sort

* Fix conflicts

* Merge remote-tracking branch 'origin/unstable' into cargo-sort
2024-12-19 05:46:03 +00:00
Pawan Dhananjay
2662dc7f8f Fix Sse client api (#6685)
* Use reqwest eventsource for get_events api

* await for Event::Open before returning stream

* fmt

* Merge branch 'unstable' into sse-client-fix

* Ignore lint
2024-12-18 00:05:58 +00:00
Age Manning
1315c94adb Unsaturate dial negotiation queue (#6711)
* Unsaturate dial-negotiation count
2024-12-17 20:10:53 +00:00
Pawan Dhananjay
d74b2d96f5 Electra alpha8 spec updates (#6496)
* Fix partial withdrawals count

* Remove get_active_balance

* Remove queue_entire_balance_and_reset_validator

* Switch to compounding when consolidating with source==target

* Queue deposit requests and apply them during epoch processing

* Fix ef tests

* Clear todos

* Fix engine api formatting issues

* Merge branch 'unstable' into electra-alpha7

* Make add_validator_to_registry more in line with the spec

* Address some review comments

* Cleanup

* Update initialize_beacon_state_from_eth1

* Merge branch 'unstable' into electra-alpha7

* Fix rpc decoding for blobs by range/root

* Fix block hash computation

* Fix process_deposits bug

* Merge branch 'unstable' into electra-alpha7

* Fix topup deposit processing bug

* Update builder api for electra

* Refactor mock builder to separate functionality

* Merge branch 'unstable' into electra-alpha7

* Address review comments

* Use copied for reference rather than cloned

* Optimise and simplify PendingDepositsContext::new

* Merge remote-tracking branch 'origin/unstable' into electra-alpha7

* Fix processing of deposits with invalid signatures

* Remove redundant code in genesis init

* Revert "Refactor mock builder to separate functionality"

This reverts commit 6d10456912.

* Revert "Update builder api for electra"

This reverts commit c5c9aca6db.

* Simplify pre-activation sorting

* Fix stale validators used in upgrade_to_electra

* Merge branch 'unstable' into electra-alpha7
2024-12-17 02:14:24 +00:00
Daniel Knopik
02cb2d68ff Enable lints for tests only running optimized (#6664)
* enable linting optimized-only tests

* fix automatically fixable or obvious lints

* fix suspicious_open_options by removing manual options

* fix `await_holding_lock`s

* avoid failing lint due to now disabled `#[cfg(debug_assertions)]`

* reduce future sizes in tests

* fix accidently flipped assert logic

* restore holding lock for web3signer download

* Merge branch 'unstable' into lint-opt-tests
2024-12-17 00:40:35 +00:00
Jimmy Chen
847c8019c7 Fix peer down-scoring behaviour when gossip blobs/columns are received after getBlobs or reconstruction (#6686)
* Fix peer disconnection when gossip blobs/columns are received after they are recieved from the EL or available via column reconstruction.
2024-12-16 05:44:14 +00:00
Lion - dapplion
1c5be34def Write range sync tests in external event-driven form (#6618)
* Write range sync tests in external event-driven form

* Fix remaining test

* Drop unused generics

* Merge branch 'unstable' into range-sync-tests

* Add reference to test author

* Use async await

* Fix failing test. Not sure how it was passing before without an EL.
2024-12-16 05:44:10 +00:00
Akihito Nakano
75d90795be Remove req_id from CustodyId (#6589)
* Remove req_id from CustodyId because it's not used
2024-12-16 05:44:06 +00:00
ethDreamer
86891e6d0f builder gas limit & some refactoring (#6583)
* Cache gas_limit

* Payload Parameters Refactor

* Enforce Proposer Gas Limit

* Fixed and Added New Tests

* Fix Beacon Chain Tests
2024-12-16 05:43:58 +00:00
Michael Sproul
a6de0d5e12 Merge remote-tracking branch 'origin/stable' into back-merge-6.0.1 2024-12-16 15:37:52 +11:00
Lion - dapplion
c92c07ff49 Track beacon processor import result metrics (#6541)
* Track beacon processor import result metrics

* Update metric name
2024-12-16 04:33:33 +00:00
Michael Sproul
0d90135047 Release v6.0.1 (#6659)
* Release v6.0.1
2024-12-16 03:03:22 +00:00
Age Manning
c3a0757ad2 Correct /nat API for libp2p (#6677)
* Fix nat API
2024-12-15 23:16:53 +00:00
Michael Sproul
f3b78889e5 Compact more when pruning states (#6667)
* Compact more when pruning states

* Merge branch 'release-v6.0.1' into compact-more
2024-12-14 08:43:00 +00:00
Lion - dapplion
775fa6730b Stuck lookup v6 (#6658)
* Fix stuck lookups if no peers on v6

* Merge branch 'release-v6.0.1' into stuck-lookup-v6
2024-12-13 05:02:10 +00:00
Povilas Liubauskas
4946343990 Fix subscribing to attestation subnets for aggregating (#6681) (#6682)
* Fix subscribing to attestation subnets for aggregating (#6681)

* Prevent scheduled subnet subscriptions from being overwritten by other subscriptions from same subnet with additional scoping by slot
2024-12-12 08:36:34 +00:00
jacobkaufmann
2b3c602b8f add HTTP API to retrieve validator IL duties 2024-12-11 20:49:34 -07:00
Michael Sproul
fc0e0ae613 Prevent reconstruction starting prematurely (#6669)
* Prevent reconstruction starting prematurely

* Simplify condition

* Merge remote-tracking branch 'origin/release-v6.0.1' into dont-start-reconstruction-early
2024-12-12 01:58:41 +00:00
Michael Sproul
b2590bcb37 Tweak reconstruction batch size (#6668)
* Tweak reconstruction batch size

* Merge branch 'release-v6.0.1' into reconstruction-batch-size
2024-12-11 22:51:46 +00:00
Michael Sproul
c5a48a9dff Merge remote-tracking branch 'origin/release-v6.0.1' into unstable 2024-12-10 15:53:35 +11:00
Age Manning
3b8254a8ec Correct flakey CI tests (#6646)
* Correct flakey CI tests

* Correct clippy

* Extend timeout for events
2024-12-10 04:24:55 +00:00
jacobkaufmann
c5c08793cc add getInclusionListV1 capability to electra readiness 2024-12-08 22:02:13 -07:00
jacobkaufmann
aca9fc2fca feat: add getInclusionListV1 to engine API client 2024-12-08 21:50:25 -07:00
hopinheimer
fec502db9f Deprecate futures ticker (#6630)
* deprecate futures-ticker

* Merge branch 'unstable' of github.com:sigp/lighthouse into deprecate-futures-timer

* Merge branch 'unstable' into deprecate-futures-timer

* making the linter happy

* remove unrequired #[allow(unused_imports)]

* fixing minor issues

* merge commit

* minor fix

* clippy changes
2024-12-03 08:56:43 +00:00
Michael Sproul
c042dc14d7 Release v6.0.0 (#6605)
* Release v6.0.0
2024-12-02 04:00:51 +00:00
Age Manning
770d677a4e Increase idle connection timeout (#6604)
* Increase idle connection timeout

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>
2024-12-02 01:44:58 +00:00
Pawan Dhananjay
f8e31f6272 Increase rpc rate limits (#6626)
* Increase rate limits for byrange requests

* Merge branch 'unstable' into reduce-blob-limits

* Update limits
2024-12-01 23:31:10 +00:00
Michael Sproul
1c8161f92b Fetch blobs from EL prior to block verification (#6600)
* Fetch blobs from EL prior to block verification

* Run fetch blobs in parallel with block import

* Merge branch 'unstable' into fetch-blobs-earlier

* Merge branch 'unstable' into fetch-blobs-earlier
2024-11-29 02:58:19 +00:00
Michael Sproul
fa6c4c02a3 Fix Rust 1.83 Clippy lints (#6629)
* Fix Rust 1.83 Clippy lints

* Cargo fmt
2024-11-29 02:23:54 +00:00
Eitan Seri-Levi
38f5f665e1 Remove error-chain dependency (#6628)
* remove error-chain dependency

* rerun CI

* rerun CI
2024-11-28 02:39:50 +00:00
jacobkaufmann
4919a3b4d7 add initial IL gossip verification 2024-11-26 20:15:00 -07:00
Age Manning
08e8b92e50 Simple Subnet Management (#6146)
* Initial temp commit

* Merge latest unstable

* First draft without tests

* Update tests for new version

* Correct comments and reviewers comments

* Merge latest unstable

* Fix errors

* Missed a comment, corrected it

* Fix lints

* Merge latest unstable

* Fix tests

* Merge latest unstable

* Reviewers comments

* Remove sync subnets from ENR on unsubscribe

* Merge branch 'unstable' into simple-peer-mapping

* Merge branch 'unstable' into simple-peer-mapping

* Merge branch 'unstable' into simple-peer-mapping

* Merge latest unstable

* Prevent clash with pin of rust_eth_kzg
2024-11-26 01:48:07 +00:00
jacobkaufmann
024dbf6efc add IL type, gossip kind, and pubsub message 2024-11-25 16:27:36 -07:00
Pawan Dhananjay
6e1945fc5d Avoid computing rpc_blob_limits multiple times (#6595)
* Compute blob rpc limits in static block

* Fix min size

* Use MainnetEthSpec in rpc tests

* Revert MainnetEthSpec; add another constant for blob size minimal
2024-11-20 23:04:45 +00:00
Pawan Dhananjay
94311c6516 Add additional metrics for idontwant (#6578)
* Add additional metrics for idontwant

* Resolve issues from review

* Fix tests

* Don't exceed capacity

* Apply suggestions from code review

Co-authored-by: João Oliveira <hello@jxs.pt>

* Return early on failure

* Add comment
2024-11-20 22:27:13 +00:00
Michael Sproul
b1e9f69460 Fix v22 schema upgrade (#6591)
* Fix v22 schema upgrade

* Ownership
2024-11-19 22:43:18 +00:00
Age Manning
8cebc87d95 Update to latest discovery (#6486)
* Upgrade discv5 to v0.8

* Rename some logs

* Improve the NAT reporting with new discv5 metrics

* Merge branch 'unstable' into discv5-v8

* Limited Cargo.lock update

* Update yanked futures-* crates
2024-11-18 22:52:23 +00:00
Michael Sproul
c5007eaa1c Deprecate eth1 and dummy-eth1 flags (#6566)
* Deprecate eth1/dummy-eth1 flags

* Update book

* Simplify + make staking conflict with disable-deposit-contract
2024-11-18 08:36:01 +00:00
Michael Sproul
9fdd53df56 Hierarchical state diffs (#5978)
* Start extracting freezer changes for tree-states

* Remove unused config args

* Add comments

* Remove unwraps

* Subjective more clear implementation

* Clean up hdiff

* Update xdelta3

* Tree states archive metrics (#6040)

* Add store cache size metrics

* Add compress timer metrics

* Add diff apply compute timer metrics

* Add diff buffer cache hit metrics

* Add hdiff buffer load times

* Add blocks replayed metric

* Move metrics to store

* Future proof some metrics

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Port and clean up forwards iterator changes

* Add and polish hierarchy-config flag

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Cleaner errors

* Fix beacon_chain test compilation

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Patch a few more freezer block roots

* Fix genesis block root bug

* Fix test failing due to pending updates

* Beacon chain tests passing

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix doc lint

* Implement DB schema upgrade for hierarchical state diffs (#6193)

* DB upgrade

* Add flag

* Delete RestorePointHash

* Update docs

* Update docs

* Implement hierarchical state diffs config migration (#6245)

* Implement hierarchical state diffs config migration

* Review PR

* Remove TODO

* Set CURRENT_SCHEMA_VERSION correctly

* Fix genesis state loading

* Re-delete some PartialBeaconState stuff

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix test compilation

* Update schema downgrade test

* Fix tests

* Fix null anchor migration

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix tree states upgrade migration (#6328)

* Towards crash safety

* Fix compilation

* Move cold summaries and state roots to new columns

* Rename StateRoots chunked field

* Update prune states

* Clean hdiff CLI flag and metrics

* Fix "staged reconstruction"

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Fix alloy issues

* Fix staged reconstruction logic

* Prevent weird slot drift

* Remove "allow" flag

* Update CLI help

* Remove FIXME about downgrade

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Remove some unnecessary error variants

* Fix new test

* Tree states archive - review comments and metrics (#6386)

* Review PR comments and metrics

* Comments

* Add anchor metrics

* drop prev comment

* Update metadata.rs

* Apply suggestions from code review

---------

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/store/src/hot_cold_store.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Clarify comment and remove anchor_slot garbage

* Simplify database anchor (#6397)

* Simplify database anchor

* Update beacon_node/store/src/reconstruct.rs

* Add migration for anchor

* Fix and simplify light_client store tests

* Fix incompatible config test

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* More metrics

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* New historic state cache (#6475)

* New historic state cache

* Add more metrics

* State cache hit rate metrics

* Fix store metrics

* More logs and metrics

* Fix logger

* Ensure cached states have built caches :O

* Replay blocks in preference to diffing

* Two separate caches

* Distribute cache build time to next slot

* Re-plumb historic-state-cache flag

* Clean up metrics

* Update book

* Update beacon_node/store/src/hdiff.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update beacon_node/store/src/historic_state_cache.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

---------

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Update database docs

* Update diagram

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Update lockbud to work with bindgen/etc

* Correct pkg name for Debian

* Remove vestigial epochs_per_state_diff

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Markdown lint

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Address Jimmy's review comments

* Simplify ReplayFrom case

* Fix and document genesis_state_root

* Typo

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into tree-states-archive

* Compute diff of validators list manually (#6556)

* Split hdiff computation

* Dedicated logic for historical roots and summaries

* Benchmark against real states

* Mutated source?

* Version the hdiff

* Add lighthouse DB config for hierarchy exponents

* Tidy up hierarchy exponents flag

* Apply suggestions from code review

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Address PR review

* Remove hardcoded paths in benchmarks

* Delete unused function in benches

* lint

---------

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Test hdiff binary format stability (#6585)

* Merge remote-tracking branch 'origin/unstable' into tree-states-archive

* Add deprecation warning for SPRP

* Update xdelta to get rid of duplicate deps

* Document test
2024-11-18 01:51:44 +00:00
Eitan Seri-Levi
654fc6acdc Additional light client metrics (#6545)
* Fix db query and add some additional metrics

* fmt

* Update beacon_node/beacon_chain/src/metrics.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update beacon_node/beacon_chain/src/metrics.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-11-15 07:09:54 +00:00
Jimmy Chen
5f053b0b6d Improving blob propagation post-PeerDAS with Decentralized Blob Building (#6268)
* Get blobs from EL.

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Avoid cloning blobs after fetching blobs.

* Address review comments and refactor code.

* Fix lint.

* Move blob computation metric to the right spot.

* Merge branch 'unstable' into das-fetch-blobs

* Merge branch 'unstable' into das-fetch-blobs

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/block_verification.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Merge branch 'unstable' into das-fetch-blobs

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs

* Gradual publication of data columns for supernodes.

* Recompute head after importing block with blobs from the EL.

* Fix lint

* Merge branch 'unstable' into das-fetch-blobs

* Use blocking task instead of async when computing cells.

* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs

* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs

* Fix semantic conflicts

* Downgrade error log.

* Merge branch 'unstable' into das-fetch-blobs

# Conflicts:
#	beacon_node/beacon_chain/src/data_availability_checker.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/network/src/network_beacon_processor/gossip_methods.rs
#	beacon_node/network/src/network_beacon_processor/mod.rs
#	beacon_node/network/src/network_beacon_processor/sync_methods.rs

* Merge branch 'unstable' into das-fetch-blobs

* Publish block without waiting for blob and column proof computation.

* Address review comments and refactor.

* Merge branch 'unstable' into das-fetch-blobs

* Fix test and docs.

* Comment cleanups.

* Merge branch 'unstable' into das-fetch-blobs

* Address review comments and cleanup

* Address review comments and cleanup

* Refactor to de-duplicate gradual publication logic.

* Add more logging.

* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs

# Conflicts:
#	Cargo.lock

* Fix incorrect comparison on `num_fetched_blobs`.

* Implement gradual blob publication.

* Merge branch 'unstable' into das-fetch-blobs

* Inline `publish_fn`.

* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs

* Gossip verify blobs before publishing

* Avoid queries for 0 blobs and error for duplicates

* Gossip verified engine blob before processing them, and use observe cache to detect duplicates before publishing.

* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs

# Conflicts:
#	beacon_node/network/src/network_beacon_processor/mod.rs

* Merge branch 'unstable' into das-fetch-blobs

* Fix invalid commitment inclusion proofs in blob sidecars created from EL blobs.

* Only publish EL blobs triggered from gossip block, and not RPC block.

* Downgrade gossip blob log to `debug`.

* Merge branch 'unstable' into das-fetch-blobs

* Merge branch 'unstable' into das-fetch-blobs

* Grammar
2024-11-15 03:34:13 +00:00