Commit Graph

2707 Commits

Author SHA1 Message Date
ethDreamer
46db30416d Implement Overflow LRU Cache for Pending Blobs (#4203)
* All Necessary Objects Implement Encode/Decode

* Major Components for LRUOverflowCache Implemented

* Finish Database Code

* Add Maintenance Methods

* Added Maintenance Service

* Persist Blobs on Shutdown / Reload on Startup

* Address Clippy Complaints

* Add (emum_behaviour = "tag") to ssz_derive

* Convert Encode/Decode Implementations to "tag"

* Started Adding Tests

* Added a ton of tests

* 1 character fix

* Feature Guard Minimal Spec Tests

* Update beacon_node/beacon_chain/src/data_availability_checker.rs

Co-authored-by: realbigsean <seananderson33@GMAIL.com>

* Address Sean's Comments

* Add iter_raw_keys method

* Remove TODOs

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-05-12 10:08:24 -04:00
realbigsean
51c4506c53 smol bugfixes and moar tests 2023-05-09 19:29:03 -04:00
Jimmy Chen
8d9c748025 Fix attestation withdrawals root mismatch (#4249)
## Issue Addressed

Addresses #4234 

## Proposed Changes

- Skip withdrawals processing in an inconsistent state replay. 
- Repurpose `StateRootStrategy`: rename to `StateProcessingStrategy` and always skip withdrawals if using `StateProcessingStrategy::Inconsistent`
- Add a test to reproduce the scenario


Co-authored-by: Jimmy Chen <jimmy@sigmaprime.io>
2023-05-09 10:48:15 +00:00
Nikita Kryuchkov
c7c51062ab Fix log on initializing external block builder (#4267)
## Issue Addressed

#4266 

## Proposed Changes

- Log `Using external block builder` instead of `Connected to external block builder` on its initialization to resolve the confusion (there's no actual connection there)

## Additional Info

The log is mentioned in builders docs, so it's changed there too.
2023-05-09 07:15:06 +00:00
ethDreamer
a22e4bf636 Implement KZG EF Tests (#4274) 2023-05-08 15:58:23 -04:00
realbigsean
6aff52c5b4 add some tests and fix a bug 2023-05-08 11:58:05 -04:00
Jack McPherson
6235e452e1 Do not attempt to resubscribe to core topics (#4271)
This commit adds a check to the networking service when handling core gossipsub topic subscription requests. If the BN is already subscribed to the core topics, we won't attempt to resubscribe.

## Issue Addressed

#4258 

## Proposed Changes

 - In the networking service, check if we're already subscribed to all of the core gossipsub topics and, if so, do nothing

## Additional Info

N/A
2023-05-08 07:15:26 +00:00
Age Manning
35ca086269 Backfill blocks only to the WSP by default (#4082)
## Limit Backfill Sync

This PR transitions Lighthouse from syncing all the way back to genesis to only syncing back to the weak subjectivity point (~ 5 months) when syncing via a checkpoint sync.

There are a number of important points to note with this PR:

- Firstly and most importantly, this PR fundamentally shifts the default security guarantees of checkpoint syncing in Lighthouse. Prior to this PR, Lighthouse could verify the checkpoint of any given chain by ensuring the chain eventually terminates at the corresponding genesis. This guarantee can still be employed via the new CLI flag --genesis-backfill which will prompt lighthouse to the old behaviour of downloading all blocks back to genesis. The new behaviour only checks the proposer signatures for the last 5 months of blocks but cannot guarantee the chain matches the genesis chain.
- I have not modified any of the peer scoring or RPC responses. Clients syncing from gensis, will downscore new Lighthouse peers that do not possess blocks prior to the WSP. This is by design, as Lighthouse nodes of this form, need a mechanism to sort through peers in order to find useful peers in order to complete their genesis sync. We therefore do not discriminate between empty/error responses for blocks prior or post the local WSP. If we request a block that a peer does not posses, then fundamentally that peer is less useful to us than other peers.
- This will make a radical shift in that the majority of nodes will no longer store the full history of the chain. In the future we could add a pruning mechanism to remove old blocks from the db also.


Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-05-05 03:49:23 +00:00
int88
6d8d212da8 use state cache to optimise historical state lookup (#4228)
## Issue Addressed

#3873

## Proposed Changes

add a cache to optimise historical state lookup.

## Additional Info

N/A


Co-authored-by: Michael Sproul <micsproul@gmail.com>
2023-05-05 00:51:57 +00:00
realbigsean
a0f6159cae make sure blobs are sent for processing after stream termination, delete copied tests 2023-05-03 12:02:02 -04:00
Akihito Nakano
edbb47dd03 Update igd to v0.12.1 (#4257)
## Issue Addressed

https://github.com/sigp/lighthouse/issues/4171

## Proposed Changes

Through [this PR](https://github.com/sbstp/rust-igd/pull/56) in rust-igd, `igd` v0.12.1 no longer panics if there is an issue while searching for a gateway. So updating igd makes lighthouse emit a helpful log instead of panicking.

## Additional Info

No CHANGELOG exists in rust-igd. 👀 Here is the commit history between v0.11.1 and v0.12.1. No breaking changes.

https://github.com/sbstp/rust-igd/compare/v0.11.1...v0.12.1
2023-05-03 04:12:14 +00:00
Jimmy Chen
2aef2db66f Un-deprecate test utils functions such as extend_chain (#4255)
## Issue Addressed

This PR un-deprecates some commonly used test util functions, e.g. `extend_chain`. Most of these were deprecated in 2020 but some of us still found them quite convenient and they're still being used a lot. If there's no issue with using them, I think we should remove the "Deprecated" comment to avoid confusion.
2023-05-03 04:12:12 +00:00
Age Manning
616bee6757 Maintain trusted peers (#4159)
## Issue Addressed
#4150 

## Proposed Changes

Maintain trusted peers in the pruning logic. ~~In principle the changes here are not necessary as a trusted peer has a max score (100) and all other peers can have at most 0 (because we don't implement positive scores). This means that we should never prune trusted peers unless we have more trusted peers than the target peer count.~~

This change shifts this logic to explicitly never prune trusted peers which I expect is the intuitive behaviour. 

~~I suspect the issue in #4150 arises when a trusted peer disconnects from us for one reason or another and then we remove that peer from our peerdb as it becomes stale. When it re-connects at some large time later, it is no longer a trusted peer.~~

Currently we do disconnect trusted peers, and this PR corrects this to maintain trusted peers in the pruning logic.

As suggested in #4150 we maintain trusted peers in the db and thus we remember them even if they disconnect from us.
2023-05-03 04:12:10 +00:00
realbigsean
9db6b39dc3 fix check on max request size (#4250) 2023-05-02 19:14:02 -04:00
realbigsean
f5facd603e fix test compilation 2023-05-02 18:39:21 -04:00
realbigsean
af15789b6f improve peer scoring 2023-05-02 18:28:45 -04:00
realbigsean
e3f4218624 error refactoring 2023-05-02 14:09:53 -04:00
realbigsean
56b2365e17 track information about peer source 2023-05-02 12:28:32 -04:00
realbigsean
8edefb7e0d make tests work for all forks 2023-05-02 10:00:07 -04:00
realbigsean
93bcd6281c some bug fixes and the start of deneb only tests 2023-04-28 15:56:54 -04:00
realbigsean
bfb5242ee3 start fixing up lookup verify error handling 2023-04-28 09:54:09 -04:00
Michael Sproul
c11638c36c Split common crates out into their own repos (#3890)
## Proposed Changes

Split out several crates which now exist in separate repos under `sigp`.

- [`ssz` and `ssz_derive`](https://github.com/sigp/ethereum_ssz)
- [`tree_hash` and `tree_hash_derive`](https://github.com/sigp/tree_hash)
- [`ethereum_hashing`](https://github.com/sigp/ethereum_hashing)
- [`ethereum_serde_utils`](https://github.com/sigp/ethereum_serde_utils)
- [`ssz_types`](https://github.com/sigp/ssz_types)

For the published crates see: https://crates.io/teams/github:sigp:crates-io?sort=recent-updates.

## Additional Info

- [x] Need to work out how to handle versioning. I was hoping to do 1.0 versions of several crates, but if they depend on `ethereum-types 0.x` that is not going to work. EDIT: decided to go with 0.5.x versions.
- [x] Need to port several changes from `tree-states`, `capella`, `eip4844` branches to the external repos.
2023-04-28 01:15:40 +00:00
realbigsean
a9b24bb9df add random blobs 2023-04-27 17:39:52 -04:00
realbigsean
185b7615a2 fix compile after merge 2023-04-27 14:30:07 -04:00
realbigsean
eadf293419 Merge branch 'deneb-free-blobs' of https://github.com/sigp/lighthouse into some-blob-reprocessing-work 2023-04-27 14:19:42 -04:00
ethDreamer
c1d47da02d Update engine_api to latest version (#4223)
* Update Engine API to Latest

* Get Mock EE Working

* Fix Mock EE

* Update Engine API Again

* Rip out get_blobs_bundle Stuff

* Fix Test Harness

* Fix Clippy Complaints

* Fix Beacon Chain Tests
2023-04-27 14:18:21 -04:00
realbigsean
d224fce084 wrap availability check error 2023-04-27 14:15:52 -04:00
realbigsean
8e9c98e8e5 make tests into macros 2023-04-27 13:02:53 -04:00
realbigsean
f2267212a5 refactor lookup tests to work with multiple forks and response types 2023-04-27 12:04:34 -04:00
realbigsean
7614abf6df fix 2023-04-26 17:55:21 -04:00
realbigsean
3c9fb78aa1 compilation fix in beacon chain tests 2023-04-26 17:28:24 -04:00
realbigsean
2cad638f41 cargo fmt 2023-04-26 16:33:08 -04:00
realbigsean
0978848caa Merge branch 'deneb-free-blobs' of https://github.com/sigp/lighthouse into some-blob-reprocessing-work 2023-04-26 14:59:48 -04:00
Justin Traglia
aa34339298 Rename to MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS (#4206)
Co-authored-by: realbigsean <sean@sigmaprime.io>
2023-04-26 14:53:06 -04:00
realbigsean
d850788278 fix after merge 2023-04-26 14:48:50 -04:00
realbigsean
13f7217172 Merge branch 'deneb-free-blobs' of https://github.com/sigp/lighthouse into some-blob-reprocessing-work 2023-04-26 14:45:50 -04:00
realbigsean
69e5e00350 renamings 2023-04-26 14:45:07 -04:00
realbigsean
4390036887 fix existing block lookup tests 2023-04-26 14:44:32 -04:00
realbigsean
46a9b3a7ed fix lints 2023-04-26 12:15:02 -04:00
realbigsean
83c3ee173f fix lints 2023-04-26 12:05:07 -04:00
realbigsean
ff58698375 fix compile in lookup tests 2023-04-26 11:49:54 -04:00
Pawan Dhananjay
a632969695 Gossip verification cleanup (#4219)
* Add ObservedBlobSidecar tests

* Add logging for tricky verification cases

* Update beacon_node/beacon_chain/src/blob_verification.rs

---------

Co-authored-by: realbigsean <seananderson33@GMAIL.com>
2023-04-26 10:44:58 -04:00
Age Manning
7456e1e8fa Separate BN for block proposals (#4182)
It is a well-known fact that IP addresses for beacon nodes used by specific validators can be de-anonymized. There is an assumed risk that a malicious user may attempt to DOS validators when producing blocks to prevent chain growth/liveness.

Although there are a number of ideas put forward to address this, there a few simple approaches we can take to mitigate this risk.

Currently, a Lighthouse user is able to set a number of beacon-nodes that their validator client can connect to. If one beacon node is taken offline, it can fallback to another. Different beacon nodes can use VPNs or rotate IPs in order to mask their IPs.

This PR provides an additional setup option which further mitigates attacks of this kind.

This PR introduces a CLI flag --proposer-only to the beacon node. Setting this flag will configure the beacon node to run with minimal peers and crucially will not subscribe to subnets or sync committees. Therefore nodes of this kind should not be identified as nodes connected to validators of any kind.

It also introduces a CLI flag --proposer-nodes to the validator client. Users can then provide a number of beacon nodes (which may or may not run the --proposer-only flag) that the Validator client will use for block production and propagation only. If these nodes fail, the validator client will fallback to the default list of beacon nodes.

Users are then able to set up a number of beacon nodes dedicated to block proposals (which are unlikely to be identified as validator nodes) and point their validator clients to produce blocks on these nodes and attest on other beacon nodes. An attack attempting to prevent liveness on the eth2 network would then need to preemptively find and attack the proposer nodes which is significantly more difficult than the default setup.

This is a follow on from: #3328 

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Paul Hauner <paul@paulhauner.com>
2023-04-26 01:12:36 +00:00
int88
90d562b3d4 add attestation inclusion distance in http api (#4148)
## Issue Addressed

#4097

## Proposed Changes

Add attestation inclusion distance in http api, extend `/lighthouse/ui/validator_metrics` to include it.

## Usage
```
curl -X POST "http://localhost:8001/lighthouse/ui/validator_metrics" -d '{"indices": [1]}' -H "Content-Type: application/json" | jq
```

```
{
  "data": {
    "validators": {
      "1": {
        "attestation_hits": 3,
        "attestation_misses": 1,
        "attestation_hit_percentage": 75,
        "attestation_head_hits": 3,
        "attestation_head_misses": 0,
        "attestation_head_hit_percentage": 100,
        "attestation_target_hits": 3,
        "attestation_target_misses": 0,
        "attestation_target_hit_percentage": 100,
        "attestation_inclusion_distance": 1
      }
    }
  }
}
```

## Additional Info

NA
2023-04-26 01:12:35 +00:00
realbigsean
14e5bfae29 fix lints 2023-04-25 18:12:16 -04:00
realbigsean
b5440f740d fix lints 2023-04-25 09:30:16 -04:00
realbigsean
9c2aceed10 fix some todos 2023-04-25 09:25:24 -04:00
realbigsean
53c0356f8d smol bugfix 2023-04-24 21:10:52 -04:00
realbigsean
b8708e38de processing peer refactor 2023-04-24 20:47:02 -04:00
realbigsean
91594adc77 refactor single block processed method 2023-04-24 20:15:45 -04:00