Commit Graph

118 Commits

Author SHA1 Message Date
Paul Hauner
8d1abce26e Bump SSZ version for larger bitfield SmallVec (#6915)
NA


  Bumps the `ethereum_ssz` version, along with other crates that share the dep.

Primarily, this give us bitfields which can store 128 bytes on the stack before allocating, rather than 32 bytes (https://github.com/sigp/ethereum_ssz/pull/38). The validator count has increase massively since we set it at 32 bytes, so aggregation bitfields (et al) now require a heap allocation. This new value of 128 should get us to ~2m active validators.
2025-03-10 08:18:33 +00:00
Lion - dapplion
0055af56b6 Unsubscribe blob topics at Fulu fork (#6932)
Addresses #6854.

PeerDAS requires unsubscribing a Gossip topic at a fork boundary. This is not possible with our current topic machinery.


  Instead of defining which topics have to be **added** at a given fork, we define the complete set of topics at a given fork. The new start of the show and key function is:

```rust
pub fn core_topics_to_subscribe<E: EthSpec>(
fork_name: ForkName,
opts: &TopicConfig,
spec: &ChainSpec,
) -> Vec<GossipKind> {
// ...
if fork_name.deneb_enabled() && !fork_name.fulu_enabled() {
// All of deneb blob topics are core topics
for i in 0..spec.blob_sidecar_subnet_count(fork_name) {
topics.push(GossipKind::BlobSidecar(i));
}
}
// ...
}
```

`core_topics_to_subscribe` only returns the blob topics if `fork < Fulu`. Then at the fork boundary, we subscribe with the new fork digest to `core_topics_to_subscribe(next_fork)`, which excludes the blob topics.

I added `is_fork_non_core_topic` to carry on to the next fork the aggregator topics for attestations and sync committee messages. This approach is future-proof if those topics ever become fork-dependent.

Closes https://github.com/sigp/lighthouse/issues/6854
2025-02-11 23:40:14 +00:00
Lion - dapplion
55d1e754b4 Subscribe to PeerDAS topics on Fulu fork (#6849)
`TODO(das)` now that PeerDAS is scheduled in a hard fork we can subscribe to its topics on the fork activation. In current stable we subscribe to PeerDAS topics as soon as the node starts if PeerDAS is scheduled.

This PR adds another todo to unsubscribe to blob topics at the fork. This other PR included solution for that, but I can include it in a separate PR
- https://github.com/sigp/lighthouse/pull/5899/files


  Include PeerDAS topics as part of Fulu fork in `fork_core_topics`.
2025-02-03 06:07:39 +00:00
João Oliveira
ddb845d503 update libp2p to 0.55 (#6889)
Updates libp2p to `0.55`.
Will address the deprecations in a subsequent PR
2025-01-31 12:21:06 +00:00
Pawan Dhananjay
4a07c08c4f Fork aware max values in rpc (#6847)
N/A


  In https://github.com/sigp/lighthouse/pull/6329 we changed `max_blobs_per_block` from a preset to a config value.
We weren't using the right value based on fork in that PR. This is a follow up PR to use the fork dependent values.

In the proces, I also updated other places where we weren't using fork dependent values from the ChainSpec.

Note to reviewer: easier to go through by commit
2025-01-29 19:42:13 +00:00
Jimmy Chen
e98209d118 Implement PeerDAS subnet decoupling (aka custody groups) (#6736)
* Implement PeerDAS subnet decoupling (aka custody groups).

* Merge branch 'unstable' into decouple-subnets

* Refactor feature testing for spec tests (#6737)

Squashed commit of the following:

commit 898d05ee17
Merge: ffbd25e2b 7e0cddef3
Author: Jimmy Chen <jchen.tc@gmail.com>
Date:   Tue Dec 24 14:41:19 2024 +1100

    Merge branch 'unstable' into refactor-ef-tests-features

commit ffbd25e2be
Author: Jimmy Chen <jchen.tc@gmail.com>
Date:   Tue Dec 24 14:40:38 2024 +1100

    Fix `SszStatic` tests for PeerDAS: exclude eip7594 test vectors when testing Electra types.

commit aa593cf35c
Author: Jimmy Chen <jchen.tc@gmail.com>
Date:   Fri Dec 20 12:08:54 2024 +1100

    Refactor spec testing for features and simplify usage.

* Fix build.

* Add input validation and improve arithmetic handling when calculating custody groups.

* Address review comments re code style consistency.

* Merge branch 'unstable' into decouple-subnets

# Conflicts:
#	beacon_node/beacon_chain/src/kzg_utils.rs
#	beacon_node/beacon_chain/src/observed_data_sidecars.rs
#	beacon_node/lighthouse_network/src/discovery/subnet_predicate.rs
#	common/eth2_network_config/built_in_network_configs/chiado/config.yaml
#	common/eth2_network_config/built_in_network_configs/gnosis/config.yaml
#	common/eth2_network_config/built_in_network_configs/holesky/config.yaml
#	common/eth2_network_config/built_in_network_configs/mainnet/config.yaml
#	common/eth2_network_config/built_in_network_configs/sepolia/config.yaml
#	consensus/types/src/chain_spec.rs

* Update consensus/types/src/chain_spec.rs

Co-authored-by: Lion - dapplion <35266934+dapplion@users.noreply.github.com>

* Merge remote-tracking branch 'origin/unstable' into decouple-subnets

* Update error handling.

* Address review comment.

* Merge remote-tracking branch 'origin/unstable' into decouple-subnets

# Conflicts:
#	consensus/types/src/chain_spec.rs

* Update PeerDAS spec tests to `1.5.0-beta.0` and fix failing unit tests.

* Merge remote-tracking branch 'origin/unstable' into decouple-subnets

# Conflicts:
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs
2025-01-15 07:40:26 +00:00
Pawan Dhananjay
587c3e2b8c Implement changes for EIP 7691 (#6803)
* Add new config options

* Use electra_enabled
2025-01-14 06:22:30 +00:00
Age Manning
770d677a4e Increase idle connection timeout (#6604)
* Increase idle connection timeout

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>
2024-12-02 01:44:58 +00:00
Eitan Seri-Levi
38f5f665e1 Remove error-chain dependency (#6628)
* remove error-chain dependency

* rerun CI

* rerun CI
2024-11-28 02:39:50 +00:00
João Oliveira
7105442840 Remove manual poll of the libp2p Swarm (#6550)
* remove manual poll for libp2p Swarm,

use tokio::select! instead
2024-10-30 11:26:26 +00:00
João Oliveira
48aa35313c lower ListenerError log level (#6544)
* lower log level of ListenerError
2024-10-29 05:26:06 +00:00
Michael Sproul
9aefb5539b Fix BlobsByRange by reverting PR6462 (#6526)
* Revert "Remove generic E from RequestId (#6462)"

This reverts commit 772929fae2.
2024-10-21 12:42:51 +00:00
Eitan Seri-Levi
d1fda938a3 Light client updates by range RPC (#6383)
* enable lc update over rpc

* resolve TODOs

* resolve merge conflicts

* move max light client updates to eth spec

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into light-client-updates-by-range-rpc

* remove ethspec dependency

* Update beacon_node/network/src/network_beacon_processor/rpc_methods.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>

* Update beacon_node/lighthouse_network/src/rpc/methods.rs

Co-authored-by: Michael Sproul <micsproul@gmail.com>
2024-10-18 02:50:51 +00:00
hopinheimer
606a113cff IDONTWANT message optimisation to cutoff for smaller messages (#6456)
* idontwant message opitmising

* requested changes and linter appeasing

* added the config cli flag

* Merge branch 'unstable' into fix/idontwant-optimise

* cli docs generated

* const declaration

* Hide extra technical cli flag

* passing ci

* Merge branch 'unstable' into fix/idontwant-optimise
2024-10-17 08:27:56 +00:00
João Oliveira
772929fae2 Remove generic E from RequestId (#6462)
* remove Ethspec from types where it's possible to do so

* remove generic E from RequestType
2024-10-16 23:05:59 +00:00
João Oliveira
17711b720e Remove duplicated connection limits checks (#6156)
* move main Behaviour to mod.rs for better readibility

and remove connection limits checks after connection has been established,

as those checks have already been done by connection limits Behaviour.

* improve logging wording wrt dial logic

when we call dial_peer we are not yet dialing but just adding the peer to the dial queue

* do not use a constant for MAX_CONNECTIONS_PER_PEER

we only use it at one place, and the function call is explicit.

* address review and re-instate connection limits checks,

but do it before the connection has been established.

* Merge branch 'unstable' of github.com:sigp/lighthouse into remove-dial-error-denied

* Merge branch 'unstable' of github.com:sigp/lighthouse into remove-dial-error-denied
2024-10-11 16:33:49 +00:00
João Oliveira
82098e1ef7 add a unique integer id to Rpc requests (#6444)
* add id to rpc requests

* rename rpc request and response types for more accurate meaning

* remove unrequired build_request function

* remove unirequired Request wrapper types and unify Outbound and Inbound Request

* add RequestId to NetworkMessage::SendResponse

,NetworkMessage::SendErrorResponse to be passed to Rpc::send_response
2024-10-01 01:36:17 +00:00
João Oliveira
5d1ff7c6f8 fix Rpc Ping sequence number (#6408)
* fix Rpc Ping sequence number

* bubble up Outbound Err's and Responses even if the peer disconnected

* send pings via Rpc from main network

* add comment to connected check

* Merge branch 'unstable' into fix-ping-seq-number
2024-09-26 14:52:03 +00:00
Michael Sproul
1447eeb40b Improve single-node testnet support and Arc NetworkConfig/ChainSpec (#6396)
* Arc ChainSpec and NetworkConfig

* Fix release tests

* Fix lint

* Merge remote-tracking branch 'origin/unstable' into single-node-testnet
2024-09-24 00:16:18 +00:00
João Oliveira
2f6ad34795 Improve rpc logic (#6400)
* update rpc imports to be explicit

* avoid exposing HandlerEvent outside RPC

it's unnecessary.

* handle Pongs at RPC handler level
2024-09-17 06:12:21 +00:00
Michael Sproul
672dcbd868 Ignore Rust 1.82 warnings about void patterns (#6357)
* Ignore Rust 1.82 warnings about void patterns
2024-09-05 02:41:19 +00:00
João Oliveira
a685dde4ad update libp2p to version 0.54 (#6249)
* update libp2p to version 0.54.0

* address review

* Merge branch 'unstable' of github.com:sigp/lighthouse into update-libp2p

* Merge branch 'update-libp2p' of github.com:sigp/lighthouse into update-libp2p
2024-09-02 16:52:18 +00:00
Pawan Dhananjay
bcff4aa825 Implement Metadatav3 (#6303)
* Add a V3 variant for metadata

* Add v3 for requests; persistence logic

* Set custody_subnets on setting metadata

* Fix tests

* Address some comments

* fmt

* Address more comments

* Fix tests

* Update metadata rpc limits

* Update method doc.
2024-08-28 06:43:12 +00:00
Lion - dapplion
f75a2cf65b PeerDAS implementation (#5683)
* 1D PeerDAS prototype: Data format and Distribution (#5050)

* Build and publish column sidecars. Add stubs for gossip.

* Add blob column subnets

* Add `BlobColumnSubnetId` and initial compute subnet logic.

* Subscribe to blob column subnets.

* Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes.

* Fix column sidecar type to use `VariableList` for data.

* Fix lint errors.

* Update types and naming to latest consensus-spec #3574.

* Fix test and some cleanups.

* Merge branch 'unstable' into das

* Merge branch 'unstable' into das

* Merge branch 'unstable' into das

# Conflicts:
#	consensus/types/src/chain_spec.rs

* Add `DataColumnSidecarsByRoot ` req/resp protocol (#5196)

* Add stub for `DataColumnsByRoot`

* Add basic implementation of serving RPC data column from DA checker.

* Store data columns in early attester cache and blobs db.

* Apply suggestions from code review

Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com>
Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com>

* Fix build.

* Store `DataColumnInfo` in database and various cleanups.

* Update `DataColumnSidecar` ssz max size and remove panic code.

---------

Co-authored-by: Eitan Seri-Levi <eserilev@gmail.com>
Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com>

* feat: add DAS KZG in data col construction (#5210)

* feat: add DAS KZG in data col construction

* refactor data col sidecar construction

* refactor: add data cols to GossipVerifiedBlockContents

* Disable windows tests for `das` branch. (c-kzg doesn't build on windows)

* Formatting and lint changes only.

* refactor: remove iters in construction of data cols

* Update vec capacity and error handling.

* Add `data_column_sidecar_computation_seconds` metric.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into das

# Conflicts:
#	.github/workflows/test-suite.yml
#	beacon_node/lighthouse_network/src/types/topics.rs

* fix: update data col subnet count from 64 to 32 (#5413)

* feat: add peerdas custody field to ENR (#5409)

* feat: add peerdas custody field to ENR

* add hash prefix step in subnet computation

* refactor test and fix possible u64 overflow

* default to min custody value if not present in ENR

* Merge branch 'unstable' into das

* Merge branch 'unstable' into das-unstable-merge-0415

# Conflicts:
#	Cargo.lock
#	beacon_node/beacon_chain/src/data_availability_checker.rs
#	beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/network/src/network_beacon_processor/mod.rs
#	beacon_node/network/src/sync/block_lookups/tests.rs
#	crypto/kzg/Cargo.toml

* Merge remote-tracking branch 'sigp/unstable' into das

* Merge remote-tracking branch 'sigp/unstable' into das

* Fix merge conflicts.

* Send custody data column to `DataAvailabilityChecker` for determining block importability (#5570)

* Only import custody data columns after publishing a block.

* Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`.

* Add custody requirement checks to `availability_cache`.

* Fix config not being passed to DAChecker and add more logging.

* Introduce `peer_das_epoch` and make blobs and columns mutually exclusive.

* Add DA filter for PeerDAS.

* Fix data availability check and use test_logger in tests.

* Fix subscribe to all data column subnets not working correctly.

* Fix tests.

* Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization.

* Remove unused data column index in `OverflowKey`.

* Fix column sidecars incorrectly produced when there are no blobs.

* Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`.

* DAS sampling on sync (#5616)

* Data availability sampling on sync

* Address @jimmygchen review

* Trigger sampling

* Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge branch 'unstable' into das

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	beacon_node/beacon_chain/src/block_verification.rs
#	beacon_node/http_api/src/publish_blocks.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/types/pubsub.rs
#	beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
#	beacon_node/store/src/hot_cold_store.rs
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/chain_spec.rs
#	consensus/types/src/eth_spec.rs

* Merge branch 'unstable' into das

* Re-process early sampling requests (#5569)

* Re-process early sampling requests

# Conflicts:
#	beacon_node/beacon_processor/src/work_reprocessing_queue.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs

* Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Add missing var

* Beta compiler fixes and small typo fixes.

* Remove duplicate method.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Merge remote-tracking branch 'sigp/unstable' into das

* Fix merge conflict.

* Add data columns by root to currently supported protocol list (#5678)

* Add data columns by root to currently supported protocol list.

* Add missing data column by roots handling.

* Merge branch 'unstable' into das

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	beacon_node/network/src/sync/block_lookups/tests.rs
#	beacon_node/network/src/sync/manager.rs

* Fix simulator tests on `das` branch (#5731)

* Bump genesis delay in sim tests as KZG setup takes longer for DAS.

* Fix incorrect YAML spacing.

* DataColumnByRange boilerplate (#5353)

* add boilerplate

* fmt

* PeerDAS custody lookup sync (#5684)

* Implement custody sync

* Lint

* Fix tests

* Fix rebase issue

* Add data column kzg verification and update `c-kzg`. (#5701)

* Add data column kzg verification and update `c-kzg`.

* Fix incorrect `Cell` size.

* Add kzg verification on rpc blocks.

* Add kzg verification on rpc data columns.

* Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (#5750)

* Fetch custody columns in range sync (#5747)

* Fetch custody columns in range sync

* Clean up todos

* Remove `BlobSidecar` construction and publish after PeerDAS activated (#5759)

* Avoid building and publishing blob sidecars after PeerDAS.

* Ignore gossip blobs with a slot greater than peer das activation epoch.

* Only attempt to verify blob count and import blobs before PeerDAS.

* #5684 review comments (#5748)

* #5684 review comments.

* Doc and message update only.

* Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s

* Make sampling tests deterministic (#5775)

* PeerDAS spec tests (#5772)

* Add get_custody_columns spec tests.

* Add kzg merkle proof spec tests.

* Add SSZ spec tests.

* Add remaining KZG tests

* Load KZG only once per process, exclude electra tests and add missing SSZ tests.

* Fix lint and missing changes.

* Ignore macOS generated file.

* Merge remote branch 'sigp/unstable' into das

* Merge remote tracking branch 'origin/unstable' into das

* Implement unconditional reconstruction for supernodes (#5781)

* Implement unconditional reconstruction for supernodes

* Move code into KzgVerifiedCustodyDataColumn

* Remove expect

* Add test

* Thanks justin

* Add withhold attack mode for interop (#5788)

* Add withhold attack mode

* Update readme

* Drop added readmes

* Undo styling changes

* Add column gossip verification and handle unknown parent block (#5783)

* Add column gossip verification and handle missing parent for columns.

* Review PR

* Fix rebase issue

* more lint issues :)

---------

Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com>

* Trigger sampling on sync events (#5776)

* Trigger sampling on sync events

* Update beacon_chain.rs

* Fix tests

* Fix tests

* PeerDAS parameter changes for devnet-0 (#5779)

* Update PeerDAS parameters to latest values.

* Lint fix

* Fix lint.

* Update hardcoded subnet count to 64 (#5791)

* Fix incorrect columns per subnet and config cleanup (#5792)

* Tidy up PeerDAS preset and config values.

* Fix broken config

* Fix DAS branch CI (#5793)

* Fix invalid syntax.

* Update cli doc. Ignore get_custody_columns test temporarily.

* Fix failing test and add verify inclusion test.

* Undo accidentally removed code.

* Only attempt reconstruct columns once. (#5794)

* Re-enable precompute table for peerdas kzg (#5795)

* Merge branch 'unstable' into das

* Update subscription filter. (#5797)

* Remove penalty for duplicate columns (expected due to reconstruction) (#5798)

* Revert DAS config for interop testing. Optimise get_custody_columns function. (#5799)

* Don't perform reconstruction for proposer node as it already has all the columns. (#5806)

* Multithread compute_cells_and_proofs (#5805)

* Multi-thread reconstruct data columns

* Multi-thread path for block production

* Merge branch 'unstable' into das

# Conflicts:
#	.github/workflows/test-suite.yml
#	beacon_node/network/src/sync/block_lookups/mod.rs
#	beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
#	beacon_node/network/src/sync/network_context.rs

* Fix CI errors.

* Move PeerDAS type-level config to configurable `ChainSpec` (#5828)

* Move PeerDAS type level config to `ChainSpec`.

* Fix tests

* Misc custody lookup improvements (#5821)

* Improve custody requests

* Type DataColumnsByRootRequestId

* Prioritize peers and load balance

* Update tests

* Address PR review

* Merge branch 'unstable' into das

* Rename deploy_block in network config (`das` branch) (#5852)

* Rename deploy_block.txt to deposit_contract_block.txt

* fmt

---------

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>

* Merge branch 'unstable' into das

* Fix CI and merge issues.

* Merge branch 'unstable' into das

# Conflicts:
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	lcli/src/main.rs

* Store data columns individually in store and caches (#5890)

* Store data columns individually in store and caches

* Implement data column pruning

* Merge branch 'unstable' into das

# Conflicts:
#	Cargo.lock

* Update reconstruction benches to newer criterion version. (#5949)

* Merge branch 'unstable' into das

# Conflicts:
#	.github/workflows/test-suite.yml

* chore: add `recover_cells_and_compute_proofs` method (#5938)

* chore: add recover_cells_and_compute_proofs method

* Introduce type alias `CellsAndKzgProofs` to address type complexity.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update `csc` format in ENR and spec tests for devnet-1 (#5966)

* Update `csc` format in ENR.

* Add spec tests for `recover_cells_and_kzg_proofs`.

* Add tests for ENR.

* Fix failing tests.

* Add protection against invalid csc value in ENR.

* Fix lint

* Fix csc encoding and decoding (#5997)

* Fix data column rpc request not being sent due to incorrect limits set. (#6000)

* Fix incorrect inbound request count causing rate limiting. (#6025)

* Merge branch 'stable' into das

# Conflicts:
#	beacon_node/network/src/sync/block_lookups/tests.rs
#	beacon_node/network/src/sync/block_sidecar_coupling.rs
#	beacon_node/network/src/sync/manager.rs
#	beacon_node/network/src/sync/network_context.rs
#	beacon_node/network/src/sync/network_context/requests.rs

* Merge remote-tracking branch 'unstable' into das

* Add kurtosis config for DAS testing (#5968)

* Add kurtosis config for DAS testing.

* Fix invalid yaml file

* Update network parameter files.

* chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (#5941)

* chore: add recover_cells_and_compute_proofs method

* chore: add rust peerdas crypto library

* chore: integrate peerdaskzg rust library into kzg crate

* chore(multi):

- update `ssz_cell_to_crypto_cell`
- update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec`

* chore(multi):

- update rest of code to handle the new crypto `Cell` type
- update test case code to no longer use the Box type

* chore: cleanup of superfluous conversions

* chore: revert c-kzg dependency back to v1

* chore: move dependency into correct order

* chore: update rust dependency

- This version includes a new method `PeerDasContext::with_num_threads`

* chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup`

* chore: cleanup exports

* chore: commit updated cargo.lock

* Update Cargo.toml

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* chore: rename dependency

* chore: update peerdas lib

- sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3

* chore: fix clippy lifetime

- Rust doesn't allow you to elide the lifetime on type aliases

* chore: cargo clippy fix

* chore: cargo fmt

* chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819)

* chore: update dependency to ignore proofs

* chore: update peerdas lib to latest

* update lib

* chore: remove empty proof parameter

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Update PeerDAS interop testnet config (#6069)

* Update interop testnet config.

* Fix typo and remove target peers

* Avoid retrying same sampling peer that previously failed. (#6084)

* Various fixes to custody range sync  (#6004)

* Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests.

* Add custody peer check before mutating `BatchInfo` to avoid inconsistent state.

* Add check to cover a case where batch is not processed while waiting for custody peers to become available.

* Fix lint and logic error

* Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`.

* Add test for `get_custody_peers_for_column`

* Revert epoch parameter refactor.

* Fall back to default custody requiremnt if peer ENR is not present.

* Add metrics and update code comment.

* Add more debug logs.

* Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible.

* Remove unused code and update docs.

* Add comment

* chore: update peerdas-kzg library (#6118)

* chore: update peerDAS lib

* chore: update library

* chore: update library to version that include "init context" benchmarks and optional validation checks

* chore: (can remove) -- Add benchmarks for init context

* Prevent continuous searchers for low-peer networks (#6162)

* Merge branch 'unstable' into das

* Fix merge conflicts

* Add cli flag to enable sampling and disable by default. (#6209)

* chore: Use reference to an array representing a blob instead of an owned KzgBlob (#6179)

* add KzgBlobRef type

* modify code to use KzgBlobRef

* clippy

* Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`.

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>

* Store computed custody subnets in PeerDB and fix custody lookup test (#6218)

* Fix failing custody lookup tests.

* Store custody subnets in PeerDB, fix custody lookup test and refactor some methods.

* Merge branch 'unstable' into das

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/block_verification_types.rs
#	beacon_node/beacon_chain/src/builder.rs
#	beacon_node/beacon_chain/src/data_availability_checker.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	beacon_node/beacon_chain/src/data_column_verification.rs
#	beacon_node/beacon_chain/src/early_attester_cache.rs
#	beacon_node/beacon_chain/src/historical_blocks.rs
#	beacon_node/beacon_chain/tests/store_tests.rs
#	beacon_node/lighthouse_network/src/discovery/enr.rs
#	beacon_node/network/src/service.rs
#	beacon_node/src/cli.rs
#	beacon_node/store/src/hot_cold_store.rs
#	beacon_node/store/src/lib.rs
#	lcli/src/generate_bootnode_enr.rs

* Fix CI failures after merge.

* Batch sampling requests by peer (#6256)

* Batch sampling requests by peer

* Fix clippy errors

* Fix tests

* Add column_index to error message for ease of tracing

* Remove outdated comment

* Fix range sync never evaluating request as finished, causing it to get stuck. (#6276)

* Merge branch 'unstable' into das-0821-merge

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/data_availability_checker.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	beacon_node/beacon_chain/src/data_column_verification.rs
#	beacon_node/beacon_chain/src/kzg_utils.rs
#	beacon_node/beacon_chain/src/metrics.rs
#	beacon_node/beacon_processor/src/lib.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/config.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/outbound.rs
#	beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/src/types/globals.rs
#	beacon_node/network/src/network_beacon_processor/mod.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/network_beacon_processor/sync_methods.rs
#	beacon_node/network/src/sync/block_lookups/common.rs
#	beacon_node/network/src/sync/block_lookups/mod.rs
#	beacon_node/network/src/sync/block_lookups/single_block_lookup.rs
#	beacon_node/network/src/sync/block_lookups/tests.rs
#	beacon_node/network/src/sync/manager.rs
#	beacon_node/network/src/sync/network_context.rs
#	consensus/types/src/data_column_sidecar.rs
#	crypto/kzg/Cargo.toml
#	crypto/kzg/benches/benchmark.rs
#	crypto/kzg/src/lib.rs

* Fix custody tests and load PeerDAS KZG instead.

* Fix ef tests and bench compilation.

* Fix failing sampling test.

* Merge pull request #6287 from jimmygchen/das-0821-merge

Merge `unstable` into `das` 20240821

* Remove get_block_import_status

* Merge branch 'unstable' into das

* Re-enable Windows release tests.

* Address some review comments.

* Address more review comments and cleanups.

* Comment out peer DAS KZG EF tests for now

* Address more review comments and fix build.

* Merge branch 'das' of github.com:sigp/lighthouse into das

* Unignore Electra tests

* Fix metric name

* Address some of Pawan's review comments

* Merge remote-tracking branch 'origin/unstable' into das

* Update PeerDAS network parameters for peerdas-devnet-2  (#6290)

* update subnet count & custody req

* das network params

* update ef tests

---------

Co-authored-by: Jimmy Chen <jchen.tc@gmail.com>
2024-08-27 04:10:22 +00:00
Lion - dapplion
ff15c78ced Implement data columns by network boilerplate (#6224)
* Implement data columns by network boilerplate

* Use correct quota values

* Address PR review

* Update currently_supported

* Merge remote-tracking branch 'sigp/unstable' into peerdas-network-boilerplate

* PR reviews

* Fix data column rpc request not being sent due to incorrect limits set. (#6000)
2024-08-13 00:16:14 +00:00
Jimmy Chen
f2fdbe7fbe Add plumbing for PeerDAS supernodes (#5050, #5409, #5570, #5966) (#6216)
* Add plumbing for peerdas supernodes (#5050, #5409, #5570, #5966)
- add cli option `--subscribe-to-all-data-columns`
- add custody subnet count to ENR, only if PeerDAS is scheduled
- subscribe to data column topics, only if PeerDAS is scheduled

Co-authored-by: Jacob Kaufmann <jacobkaufmann18@gmail.com>

* Merge branch 'unstable' into das-supernode

* Update CLI docs.

* Merge branch 'unstable' into das-supernode

* Fix fork epoch comparison with `FAR_FUTURE_EPOCH`.

* Merge branch 'unstable' into das-supernode

* Hide `--subscribe-all-data-column-subnets` flag and update help.

* Fix docs only

* Merge branch 'unstable' into das-supernode
2024-08-12 02:31:21 +00:00
João Oliveira
28e3b86bee limit dial concurrency on Swarm (#6184)
* limit dial concurrency on Swarm

* Merge branch 'unstable' into set-dial-concurrency
2024-07-26 01:21:07 +00:00
Jimmy Chen
4e5a363a4f Add DataColumnSidecar gossip topic and message handling (#6147)
* Add `DataColumnSidecar` gossip topic and verification (#5050 and #5783).

* Remove gossip verification changes (#5783).

* Merge branch 'unstable' into data-column-gossip

# Conflicts:
#	beacon_node/beacon_chain/src/data_column_verification.rs
#	beacon_node/beacon_chain/src/lib.rs

* Add gossip cache timeout for data columns. Rename data column metrics for consistency.

* Remove usage of `unimplemented!` and address review comments.

* Remove unnused `GossipDataColumnError` variants and address review comments.

* Merge branch 'unstable' into data-column-gossip

* Update Cargo.lock

* Arc `ChainSpec` in discovery to avoid performance regression when needing to clone it repeatedly.
2024-07-25 06:05:18 +00:00
dknopik
a2ab26c327 Use then instead of then_some when checking upnp_enabled to avoid useless UPnP query (#6170)
* Use `then` instead of `then_some` when checking `upnp_enabled` to avoid useless UPnP query
2024-07-25 03:39:02 +00:00
João Oliveira
d72dca5e79 level down libp2p quic accept and close errors (#6148)
* ignore libp2p-quic accept and close errors
2024-07-23 04:29:48 +00:00
João Oliveira
a59a61fef9 Remove generic Id param from RequestId (#6032)
* rename RequestId's for better context,

and move them to lighthouse_network crate.

* remove unrequired generic AppReqId from RequestID
2024-07-08 23:56:14 +00:00
Age Manning
22fe0a6622 Allow all RPC messages on disconnect (#5876)
* Permit rpc messages on disconnect

* Use or instead of and
2024-06-07 00:56:40 +00:00
Age Manning
7b48b0b4a7 Errors for all RPC Requests (#5867)
* Return and error if peer has disconnected

* Report errors for rate limited requests

* Code improvement

* Bump rust version to 1.78

* Downgrade to 1.77

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: João Oliveira <hello@jxs.pt>

* fix fmt

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into rpc-peer-disconnect-error

* update lockfile
2024-06-07 00:12:45 +00:00
Lion - dapplion
7073242ccc Suppress RPC Error disconnect log (#5802)
* Suppress RPC Error disconnect log
2024-05-23 14:34:49 +00:00
Lion - dapplion
b87c36ac0e Report RPC Errors to the application on peer disconnections (#5680)
* Report RPC Errors to the application on peer disconnections

Co-authored-by: Age Manning <Age@AgeManning.com>

* Expect RPCError::Disconnect to fail ongoing requests

* Drop lookups after peer disconnect and not awaiting events

* Allow RPCError disconnect through network service

* Update beacon_node/lighthouse_network/src/service/mod.rs

Co-authored-by: Age Manning <Age@AgeManning.com>

* Merge branch 'unstable' into rpc-error-on-disconnect
2024-05-06 17:18:47 +00:00
Age Manning
59753f5fed Improve ENR updates (#5483)
* Improve ENR updates

* forever fmt

* Appease my old friend clippy

* Merge network unstable
2024-05-01 08:40:19 +00:00
Jimmy Chen
d30ba976a1 Use the mesh_n value from NetworkLoad for PeerScoreSettings (#5013)
* Build `gs_config` and use the `mesh_n` config for `PeerScoreSettings`

* Remove unused imports

* Merge branch 'unstable' into peer-score-settings

# Conflicts:
#	beacon_node/lighthouse_network/src/config.rs
#	beacon_node/lighthouse_network/src/service/gossipsub_scoring_parameters.rs
#	beacon_node/lighthouse_network/src/service/mod.rs
2024-04-11 20:14:15 +00:00
GeemoCandama
32be063f0f Support LightClientFinalityUpdate and LightClientOptimisticUpdate rpcs (#3849)
* add light client optimistic and finality update rpc

* Arc the updates in the response

* add conditional advertisement for both LightClientOptimisticUpdate and LightClientFinalityUpdate

* alter display for inboundrequest light client optimistic and finality updates

* remove LightClientOptimistic/FinalityReuest struct and some minor fixes

* rebase

* failing rpc_test for LightClientBootstrap and beginning of MockLib2pLightClient

* minor change

* added MockRPCHandler by importing everything except OutboundRequest. Need to implement the ConnectionHandler trait now should be copy pastable

* almost there but ran into issue where needed to implement BaseOutboundRequest.

* failing but running with a light client service of sorts

* small test change

* changed Protocol::LightClientBootstrap response limit

* deleted some stuff from ConnectionHandler Implementation for the mock light client if you need to make something with multiple requests work maybe check here

* deleted purging expired inbound/outbound streams code

* deleted drive inbound streams that need to be processed

* removed unused imports

* made things private again

* deleted inject_fully_negotiated_inbound

* made more things private again

* more

* turned the logger off in the test

* added failing test for new rpc

* add rate limit for new rpcs

* change InboundUpgrade function to use new rpcs. fmt. add test for LightClientFinalityUpdate

* rebasing fix

* add LightClientUpdate to handle_rpc functions

* added context bytes

* fmt

* use correct unsed_tcp4_port function

* fix for recent config changes and adding context_bytes for the light client protocols

* fix clippy complaint

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/beacon_processor/src/lib.rs
#	beacon_node/lighthouse_network/src/peer_manager/mod.rs
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/config.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/outbound.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/rpc/rate_limiter.rs
#	beacon_node/lighthouse_network/src/rpc/self_limiter.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/lighthouse_network/tests/rpc_tests.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs

* Error handling updates and various cleanups.

* Moar minor clean ups.

* Do not ban peer for rate limiting light client requests

* Merge branch 'unstable' into lc-reqresp. Also removed the mock light client tests to make it compile (See #4940).

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/methods.rs
#	beacon_node/lighthouse_network/src/rpc/mod.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	beacon_node/lighthouse_network/tests/common/mod.rs
#	beacon_node/network/src/network_beacon_processor/rpc_methods.rs
#	beacon_node/network/src/router.rs
#	consensus/types/src/light_client_bootstrap.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Remove unnecessary changes

* Add missing light client queue handling.

* Merge branch 'unstable' into lc-reqresp

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs
#	consensus/types/src/light_client_finality_update.rs
#	consensus/types/src/light_client_optimistic_update.rs

* Add context bytes for light client RPC responses.

* Add RPC limits for light client object.

* Fix lint

* Fix incorrect light client max size computation.

* Merge branch 'unstable' into lc-reqresp

# Conflicts:
#	beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs
#	beacon_node/lighthouse_network/src/rpc/protocol.rs
#	beacon_node/lighthouse_network/src/service/api_types.rs

* Remove unwanted local changes.

* Merge branch 'unstable' into lc-reqresp

* Replace `unimplemented` electra code path with deneb values.
2024-04-09 07:23:39 +00:00
Mac L
969d12dc6f Use E for EthSpec globally (#5264)
* Use `E` for `EthSpec` globally

* Fix tests

* Merge branch 'unstable' into e-ethspec

* Merge branch 'unstable' into e-ethspec

# Conflicts:
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/http.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/execution_layer/src/test_utils/handle_rpc.rs
#	beacon_node/store/src/partial_beacon_state.rs
#	consensus/types/src/beacon_block.rs
#	consensus/types/src/beacon_block_body.rs
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/config_and_preset.rs
#	consensus/types/src/execution_payload.rs
#	consensus/types/src/execution_payload_header.rs
#	consensus/types/src/light_client_optimistic_update.rs
#	consensus/types/src/payload.rs
#	lcli/src/parse_ssz.rs
2024-04-02 15:12:25 +00:00
Pawan Dhananjay
c909941ca1 Bump duplicate cache time (#5493)
* Bump seen_ttl for gossipsub duplicate cache
2024-03-27 17:31:51 +00:00
João Oliveira
59ef564b1d Move gossipsub into a separate crate (#5401)
* move gossipsub into a separate crate

* Merge branch 'unstable' of github.com:sigp/lighthouse into separate-gossipsub

* address review 2

* clippy beta

* update logging to log gossipsub logs
2024-03-26 03:10:59 +00:00
João Oliveira
6edf031490 disable libp2p upnp (#5449)
* disable libp2p upnp

when passing --disable-upnp cli flag
2024-03-21 01:13:03 +00:00
Jonas Bostoen
641f6be3f0 Explicit peers (#5333)
* Merge branch 'unstable' into feature/explicit-peers

* Merge latest unstable

* refactor: remove explicit-peers flag, mark trusted peers as explicit instead

* feat(beacon_node): add explicit peers to GossipSub, mark as trusted

* feat(beacon_node): add explicit peers cli + config
2024-03-07 22:22:39 +00:00
Age Manning
85c3204d70 Correct the metrics for topic subscriptions (#5344)
* Handle fork boundaries

* Merge latest unstable

* Topic subscription fix
2024-03-07 12:32:21 +00:00
João Oliveira
abd99652b4 remove nat module and use libp2p upnp (#4840)
* remove nat module and use libp2p upnp

* update Cargo.lock

* remove no longer used dependencies

* restore nat module refactored

* log successful mapping

* only activate upnp if config enabled

reduce logs to debug!

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* address review

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* Merge branch 'unstable' of https://github.com/sigp/lighthouse into libp2p-nat

* address review
2024-02-27 07:29:18 +00:00
Michael Sproul
7c23625193 Quieten gossip republish logs (#5235)
* Quieten gossip republish logs
2024-02-15 04:18:23 +00:00
Age Manning
4db84de563 Improve network parameters (#5177)
* Modify network parameters for current mainnet conditions
2024-02-08 02:40:47 +00:00
Age Manning
853042746b Downgrade gossipsub duplicate logs (#5163)
* Downgrade duplicate publish logs

* Maintain backwards compatiblity, deprecate flag

* The tests had to go, because there's no config to test against

* Update help_bn.md
2024-02-06 07:24:01 +00:00
Age Manning
f02189c86a Prevent QUIC logs when quic is disabled (#5071)
* Prevent logs and dialing quic multiaddrs when not supported

* Merge latest unstable
2024-01-31 18:11:49 +00:00
Age Manning
4273004bd9 Add gossipsub as a Lighthouse behaviour (#5066)
* Move gossipsub as a lighthouse behaviour

* Update dependencies, pin to corrected libp2p version

* Merge latest unstable

* Fix test

* Remove unused dep

* Fix cargo.lock

* Re-order behaviour, pin upstream libp2p

* Pin discv5 to latest version
2024-01-31 17:32:31 +00:00