Commit Graph

778 Commits

Author SHA1 Message Date
realbigsean
e503c8e002 Merge branch 'deneb-free-blobs' of https://github.com/sigp/lighthouse into some-blob-reprocessing-work 2023-06-02 08:19:58 -04:00
realbigsean
28b92cde12 pr feedback 2023-06-01 12:38:53 -04:00
realbigsean
f31c551fce Merge branch 'some-blob-reprocessing-work' of https://github.com/realbigsean/lighthouse into some-blob-reprocessing-work 2023-06-01 08:50:29 -04:00
realbigsean
8c5c92be06 fix bug in matching blocks and blobs in range sync 2023-06-01 08:50:03 -04:00
realbigsean
4ba384990d Update beacon_node/network/src/sync/network_context.rs
Co-authored-by: Divma <26765164+divagant-martian@users.noreply.github.com>
2023-06-01 08:29:12 -04:00
Jimmy Chen
70c4ae35ab Merge branch 'unstable' into deneb-free-blobs
# Conflicts:
#	.github/workflows/docker.yml
#	.github/workflows/local-testnet.yml
#	.github/workflows/test-suite.yml
#	Cargo.lock
#	Cargo.toml
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/builder.rs
#	beacon_node/beacon_chain/src/test_utils.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/network/src/beacon_processor/mod.rs
#	beacon_node/network/src/beacon_processor/worker/gossip_methods.rs
#	beacon_node/network/src/sync/backfill_sync/mod.rs
#	beacon_node/store/src/config.rs
#	beacon_node/store/src/hot_cold_store.rs
#	common/eth2_network_config/Cargo.toml
#	consensus/ssz/src/decode/impls.rs
#	consensus/ssz_derive/src/lib.rs
#	consensus/ssz_derive/tests/tests.rs
#	consensus/ssz_types/src/serde_utils/mod.rs
#	consensus/tree_hash/src/impls.rs
#	consensus/tree_hash/src/lib.rs
#	consensus/types/Cargo.toml
#	consensus/types/src/beacon_state.rs
#	consensus/types/src/chain_spec.rs
#	consensus/types/src/eth_spec.rs
#	consensus/types/src/fork_name.rs
#	lcli/Cargo.toml
#	lcli/src/main.rs
#	lcli/src/new_testnet.rs
#	scripts/local_testnet/el_bootnode.sh
#	scripts/local_testnet/genesis.json
#	scripts/local_testnet/geth.sh
#	scripts/local_testnet/setup.sh
#	scripts/local_testnet/start_local_testnet.sh
#	scripts/local_testnet/vars.env
#	scripts/tests/doppelganger_protection.sh
#	scripts/tests/genesis.json
#	scripts/tests/vars.env
#	testing/ef_tests/Cargo.toml
#	validator_client/src/block_service.rs
2023-05-30 22:44:05 +10:00
realbigsean
944c9736e7 penalize blob peer on invalid blobs 2023-05-23 16:05:21 -04:00
realbigsean
98d7a1014f get tests compiling 2023-05-23 12:10:54 -04:00
realbigsean
8ea98c596c merge unknown parent messages before current slot lookup 2023-05-23 11:33:00 -04:00
realbigsean
c59a3fcf42 retry on invalid block/blob 2023-05-22 11:02:41 -04:00
realbigsean
c7aad773a8 lots of refactoring 2023-05-19 17:47:44 -04:00
Paul Hauner
01ae37ac37 Add more metrics for tracking sync messages (#4308)
## Issue Addressed

NA

## Proposed Changes

Adds metrics to track validators that are submitting equivocating (but not slashable) sync messages. This follows on from some research we've been doing in a separate fork of LH.

## Additional Info

@jimmygchen and @michaelsproul have already run their eyes over this so it should be easy to get into v4.2.0, IMO.
2023-05-19 05:13:07 +00:00
realbigsean
bef63e42f7 compile after merge 2023-05-18 16:42:13 -04:00
realbigsean
4dd1184968 Merge branch 'deneb-free-blobs' of https://github.com/sigp/lighthouse into some-blob-reprocessing-work 2023-05-18 15:27:59 -04:00
realbigsean
542e3c16db add tests and fix some things 2023-05-18 15:20:56 -04:00
Paul Hauner
7c0b2755c2 Don't requeue already-known RPC blocks (#4214)
## Issue Addressed

NA

## Proposed Changes

Adds an additional check to a feature introduced in #4179 to prevent us from re-queuing already-known blocks that could be rejected immediately.

## Additional Info

Ideally this would have been included in v4.1.0, however we came across it too late to release it safely. We decided that the safest path forward is to release *without* this check and then patch it in the next version. The lack of this check should only result in a very minor performance impact (the impact is totally negligible in my assessment).
2023-05-15 07:22:04 +00:00
Paul Hauner
714ed53839 Add a flag for storing invalid blocks (#4194)
## Issue Addressed

NA

## Proposed Changes

Adds a flag to store invalid blocks on disk for teh debugz. Only *some* invalid blocks are stored, those which:

- Were received via gossip (rather than RPC, for instance)
    - This keeps things simple to start with and should capture most blocks.
- Passed gossip verification
    - This reduces the ability for random people to fill up our disk. A proposer signature is required to write something to disk.

## Additional Info

It's possible that we'll store blocks that aren't necessarily invalid, but we had an internal error during verification. Those blocks seem like they might be useful sometimes.
2023-05-15 07:22:03 +00:00
ethDreamer
46db30416d Implement Overflow LRU Cache for Pending Blobs (#4203)
* All Necessary Objects Implement Encode/Decode

* Major Components for LRUOverflowCache Implemented

* Finish Database Code

* Add Maintenance Methods

* Added Maintenance Service

* Persist Blobs on Shutdown / Reload on Startup

* Address Clippy Complaints

* Add (emum_behaviour = "tag") to ssz_derive

* Convert Encode/Decode Implementations to "tag"

* Started Adding Tests

* Added a ton of tests

* 1 character fix

* Feature Guard Minimal Spec Tests

* Update beacon_node/beacon_chain/src/data_availability_checker.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Address Sean's Comments

* Add iter_raw_keys method

* Remove TODOs

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-05-12 10:08:24 -04:00
realbigsean
51c4506c53 smol bugfixes and moar tests 2023-05-09 19:29:03 -04:00
realbigsean
6aff52c5b4 add some tests and fix a bug 2023-05-08 11:58:05 -04:00
Jack McPherson
6235e452e1 Do not attempt to resubscribe to core topics (#4271)
This commit adds a check to the networking service when handling core gossipsub topic subscription requests. If the BN is already subscribed to the core topics, we won't attempt to resubscribe.

## Issue Addressed

#4258 

## Proposed Changes

 - In the networking service, check if we're already subscribed to all of the core gossipsub topics and, if so, do nothing

## Additional Info

N/A
2023-05-08 07:15:26 +00:00
Age Manning
35ca086269 Backfill blocks only to the WSP by default (#4082)
## Limit Backfill Sync

This PR transitions Lighthouse from syncing all the way back to genesis to only syncing back to the weak subjectivity point (~ 5 months) when syncing via a checkpoint sync.

There are a number of important points to note with this PR:

- Firstly and most importantly, this PR fundamentally shifts the default security guarantees of checkpoint syncing in Lighthouse. Prior to this PR, Lighthouse could verify the checkpoint of any given chain by ensuring the chain eventually terminates at the corresponding genesis. This guarantee can still be employed via the new CLI flag --genesis-backfill which will prompt lighthouse to the old behaviour of downloading all blocks back to genesis. The new behaviour only checks the proposer signatures for the last 5 months of blocks but cannot guarantee the chain matches the genesis chain.
- I have not modified any of the peer scoring or RPC responses. Clients syncing from gensis, will downscore new Lighthouse peers that do not possess blocks prior to the WSP. This is by design, as Lighthouse nodes of this form, need a mechanism to sort through peers in order to find useful peers in order to complete their genesis sync. We therefore do not discriminate between empty/error responses for blocks prior or post the local WSP. If we request a block that a peer does not posses, then fundamentally that peer is less useful to us than other peers.
- This will make a radical shift in that the majority of nodes will no longer store the full history of the chain. In the future we could add a pruning mechanism to remove old blocks from the db also.


Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-05-05 03:49:23 +00:00
realbigsean
a0f6159cae make sure blobs are sent for processing after stream termination, delete copied tests 2023-05-03 12:02:02 -04:00
Akihito Nakano
edbb47dd03 Update igd to v0.12.1 (#4257)
## Issue Addressed

https://github.com/sigp/lighthouse/issues/4171

## Proposed Changes

Through [this PR](https://github.com/sbstp/rust-igd/pull/56) in rust-igd, `igd` v0.12.1 no longer panics if there is an issue while searching for a gateway. So updating igd makes lighthouse emit a helpful log instead of panicking.

## Additional Info

No CHANGELOG exists in rust-igd. 👀 Here is the commit history between v0.11.1 and v0.12.1. No breaking changes.

https://github.com/sbstp/rust-igd/compare/v0.11.1...v0.12.1
2023-05-03 04:12:14 +00:00
Age Manning
616bee6757 Maintain trusted peers (#4159)
## Issue Addressed
#4150 

## Proposed Changes

Maintain trusted peers in the pruning logic. ~~In principle the changes here are not necessary as a trusted peer has a max score (100) and all other peers can have at most 0 (because we don't implement positive scores). This means that we should never prune trusted peers unless we have more trusted peers than the target peer count.~~

This change shifts this logic to explicitly never prune trusted peers which I expect is the intuitive behaviour. 

~~I suspect the issue in #4150 arises when a trusted peer disconnects from us for one reason or another and then we remove that peer from our peerdb as it becomes stale. When it re-connects at some large time later, it is no longer a trusted peer.~~

Currently we do disconnect trusted peers, and this PR corrects this to maintain trusted peers in the pruning logic.

As suggested in #4150 we maintain trusted peers in the db and thus we remember them even if they disconnect from us.
2023-05-03 04:12:10 +00:00
realbigsean
9db6b39dc3 fix check on max request size (#4250) 2023-05-02 19:14:02 -04:00
realbigsean
f5facd603e fix test compilation 2023-05-02 18:39:21 -04:00
realbigsean
af15789b6f improve peer scoring 2023-05-02 18:28:45 -04:00
realbigsean
e3f4218624 error refactoring 2023-05-02 14:09:53 -04:00
realbigsean
56b2365e17 track information about peer source 2023-05-02 12:28:32 -04:00
realbigsean
8edefb7e0d make tests work for all forks 2023-05-02 10:00:07 -04:00
realbigsean
93bcd6281c some bug fixes and the start of deneb only tests 2023-04-28 15:56:54 -04:00
realbigsean
bfb5242ee3 start fixing up lookup verify error handling 2023-04-28 09:54:09 -04:00
Michael Sproul
c11638c36c Split common crates out into their own repos (#3890)
## Proposed Changes

Split out several crates which now exist in separate repos under `sigp`.

- [`ssz` and `ssz_derive`](https://github.com/sigp/ethereum_ssz)
- [`tree_hash` and `tree_hash_derive`](https://github.com/sigp/tree_hash)
- [`ethereum_hashing`](https://github.com/sigp/ethereum_hashing)
- [`ethereum_serde_utils`](https://github.com/sigp/ethereum_serde_utils)
- [`ssz_types`](https://github.com/sigp/ssz_types)

For the published crates see: https://crates.io/teams/github:sigp:crates-io?sort=recent-updates.

## Additional Info

- [x] Need to work out how to handle versioning. I was hoping to do 1.0 versions of several crates, but if they depend on `ethereum-types 0.x` that is not going to work. EDIT: decided to go with 0.5.x versions.
- [x] Need to port several changes from `tree-states`, `capella`, `eip4844` branches to the external repos.
2023-04-28 01:15:40 +00:00
realbigsean
a9b24bb9df add random blobs 2023-04-27 17:39:52 -04:00
realbigsean
eadf293419 Merge branch 'deneb-free-blobs' of https://github.com/sigp/lighthouse into some-blob-reprocessing-work 2023-04-27 14:19:42 -04:00
ethDreamer
c1d47da02d Update engine_api to latest version (#4223)
* Update Engine API to Latest

* Get Mock EE Working

* Fix Mock EE

* Update Engine API Again

* Rip out get_blobs_bundle Stuff

* Fix Test Harness

* Fix Clippy Complaints

* Fix Beacon Chain Tests
2023-04-27 14:18:21 -04:00
realbigsean
d224fce084 wrap availability check error 2023-04-27 14:15:52 -04:00
realbigsean
8e9c98e8e5 make tests into macros 2023-04-27 13:02:53 -04:00
realbigsean
f2267212a5 refactor lookup tests to work with multiple forks and response types 2023-04-27 12:04:34 -04:00
realbigsean
69e5e00350 renamings 2023-04-26 14:45:07 -04:00
realbigsean
4390036887 fix existing block lookup tests 2023-04-26 14:44:32 -04:00
realbigsean
46a9b3a7ed fix lints 2023-04-26 12:15:02 -04:00
realbigsean
83c3ee173f fix lints 2023-04-26 12:05:07 -04:00
realbigsean
ff58698375 fix compile in lookup tests 2023-04-26 11:49:54 -04:00
Age Manning
7456e1e8fa Separate BN for block proposals (#4182)
It is a well-known fact that IP addresses for beacon nodes used by specific validators can be de-anonymized. There is an assumed risk that a malicious user may attempt to DOS validators when producing blocks to prevent chain growth/liveness.

Although there are a number of ideas put forward to address this, there a few simple approaches we can take to mitigate this risk.

Currently, a Lighthouse user is able to set a number of beacon-nodes that their validator client can connect to. If one beacon node is taken offline, it can fallback to another. Different beacon nodes can use VPNs or rotate IPs in order to mask their IPs.

This PR provides an additional setup option which further mitigates attacks of this kind.

This PR introduces a CLI flag --proposer-only to the beacon node. Setting this flag will configure the beacon node to run with minimal peers and crucially will not subscribe to subnets or sync committees. Therefore nodes of this kind should not be identified as nodes connected to validators of any kind.

It also introduces a CLI flag --proposer-nodes to the validator client. Users can then provide a number of beacon nodes (which may or may not run the --proposer-only flag) that the Validator client will use for block production and propagation only. If these nodes fail, the validator client will fallback to the default list of beacon nodes.

Users are then able to set up a number of beacon nodes dedicated to block proposals (which are unlikely to be identified as validator nodes) and point their validator clients to produce blocks on these nodes and attest on other beacon nodes. An attack attempting to prevent liveness on the eth2 network would then need to preemptively find and attack the proposer nodes which is significantly more difficult than the default setup.

This is a follow on from: #3328 

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-04-26 01:12:36 +00:00
realbigsean
14e5bfae29 fix lints 2023-04-25 18:12:16 -04:00
realbigsean
b5440f740d fix lints 2023-04-25 09:30:16 -04:00
realbigsean
9c2aceed10 fix some todos 2023-04-25 09:25:24 -04:00
realbigsean
53c0356f8d smol bugfix 2023-04-24 21:10:52 -04:00