N/A
Implement range sync in gloas.
Basically requests blocks and payloads post gloas from the same peer, couples them and sends it for processing.
Does not change sync much at all other than adding the machinery for payloads by range requests.
Main changes are:
`RangeSyncBlock` which used to be a struct is an enum to account for the Gloas case. This allows a clear separation between gloas and pre-gloas code.
`AvailableBlockData` now has a `BlockInEnvelope` variant. This is to clearly indicate the post gloas case. I feel this is simpler to follow compared to `NoData` variant.
Tries to extract post gloas logic into its own functions so that there is minimal logic change in mainnet range sync behaviour.
This is meant as a stable base on which we can iterate further to make range sync cleaner and for unblocking range sync support on devnet. Some ideas for later is removing the retry mechanism in favour of delegating column fetching to lookup sync which can be done post #9155 and batch signature verifying envelopes.
Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
N/A
Currently, we have `EnvelopeError` having a `ImportError` wrapping a `BlockError`. I feel this is extremely unintuitive because most of the envelope processing functions can simply return an `EnvelopeError` that makes sense in the function's context. It revealed further ugliness when implementing range sync in #9362
This PR does 2 main things:
1. Removes `ImportError(BlockError)` variant
2. Adds `EnvelopeError(EnvelopeError)` variant to a `BlockError`.
I feel this is more natural as there can be envelope errors when we try importing a Block but envelope errors can be contained to just envelope related errors.
The main blocker to doing this was `PayloadVerificationHandle` returning a `BlockError`. It uses a very small subset of `BlockError` which I extracted to its own error type which can be converted into both a BlockError and EnvelopeError.
This allows us to keep most of the pure envelope processing functions to just return EnvelopeErrors while we convert it to a `BlockError` only in import paths where we need to return a consolidated `BlockError`.
Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
See related issue: https://github.com/ethpandaops/dora/pull/713
When LH emits a `head` event the block isn't written to disk yet. Some upstream consumers may expect that after a `head` event that the block should be queryable via the beacon api. This PR falls back to fetching the block from the early attester cache if it wasn't found in the store. This should ensure that a block is always queryable immediately after a `head` event is emitted.
Additionally I noticed that when serving columns we always default to using the store. We already have `get_data_columns_checking_all_caches ` which tries the da cache, then the store and finally the early attester cache.
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
#8314 left a few ugly potentially panicking location behind - all of them believed to be unreachable, but this PR fixes them regardless for good hygiene.
Update to `ethereum_ssz 0.10.4` for two new helpers: `not_inplace` and `clone_zeroed`.
Remove remaining `expect` and `todo!` in favour of these helpers and one new fallible (but practically infallible) method.
Co-Authored-By: Daniel Knopik <daniel@dknopik.de>
- PR https://github.com/sigp/lighthouse/pull/9305 wants to store PTCs in the committee cache.
BUT the http API route wants to use the committee cache and insert historical committees (i.e. given state at epoch 1000, compute and store the committee for epoch 900).
If we want a single cache to serve both use cases we need to:
- Have entries in the committee cache that have no PTC: Makes reading PTCs from the cache not deterministic
- Compute historical PTC: A bunch of complicated code that's useless
Instead we can add a separate cache for the API, very simple one, that caches committees only. And have the one in the beacon chain compute and cache PTCs always.
### Performance impact
Slightly additional memory cost for users of the `beacon/states/committees` route. Caching is almost equivalent, except for queries of recent committees that may already exist in the beacon chain's committee cache.
### AI disclousure
This PR was written by hand 90%. Claude fixed some warp type issues
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
While working on this code in another branch I noticed we had this messy, complicated and incorrect code about SPRP (slots-per-restore-point), which is no longer a relevant concept since the introduction of hot state diffs.
In the name of simplicity, I've removed any kind of hinting here in favour of a simple out of bounds error. The benefit of adding complex hinting code (which is not tested) to such a function is not worth it IMO. Users will work it out (or ask) if we just tell them their request is out of bounds.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
In Gloas, beacon blocks are imported into fork choice immediately - the payload envelope and data columns arrive
separately. KZG commitments moved from the column sidecar into the execution payload bid, so the existing
`DataAvailabilityChecker` (which assumes block and data are coupled) can't be used for Gloas.
* Introduced `PendingPayloadCache` to keep track of payload and data columns per block root.
* Added gossip column verification
* Added support for Gloas data column reconstruction
* Payload envelope verification simplified: removed `MaybeAvailableEnvelope`, `ExecutedEnvelope`, `EnvelopeImportData`
Not yet implemented (tracked with TODOs):
- Proper lookup sync for Gloas columns arriving before blocks
- Partial column merging for Gloas
- Moving `load_gloas_payload_bid` disk reads off the async runtime
- Backfill/range sync for Gloas
Based on @eserilev's PR and work in progress. See also #9202 for verification.
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com>
Co-Authored-By: Daniel Knopik <daniel@dknopik.de>
Co-Authored-By: Daniel Knopik <107140945+dknopik@users.noreply.github.com>
Co-Authored-By: dapplion <35266934+dapplion@users.noreply.github.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Test helpers `add_attested_block_at_slot` and `add_attested_blocks_at_slot` accepted `state_root` argument which was computed before applying the block.
Co-Authored-By: hopinheimer <knmanas6@gmail.com>
Part of #8828 for the stateful path and helps align gloas `produceBlockV4` with beacon-APIs [PR](https://github.com/ethereum/beacon-APIs/pull/580)
- Plumb `include_payload` query through the handler. Ignored for now since stateless mode isn't wired up yet
- Add `execution_payload_included` metadata field + `Eth-Execution-Payload-Included` header per spec. Both `false` until stateless lands
- Drop the `{builder_index}` segment from the envelope GET URL since no longer included in spec
Co-Authored-By: shane-moore <skm1790@gmail.com>
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Addresses issue #9220
The `payload_attestation_data` endpoint returns 400 when no block has been received for the requested slot. This causes the VC to log at CRIT level for what is expected behaviour per spec: validators should simply not submit a payload attestation when no block has been seen.
- Return 404 (Not Found) instead of 400 from `payload_attestation_data` when no block exists for the slot. This is consistent with other beacon api endpoints.
- Downgrade the VC log from `crit` to `debug` when a 503 is received, since this is an expected no-op per spec.
- Add `BlockNotFound` rejection type to `warp_utils`.
- Add a test asserting the 404 response for an empty slot.
Co-Authored-By: Josh King <josh@sigmaprime.io>
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Allow for the vc to submit its proposer preferences to the network
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
We are not submitting ptc votes that we produce to our lcoal ptc op pool. So when we are the block producer we don't include our own ptc votes!
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Fixes a bug where a proposer votes payload missing on its own block.
The payload is published to the network but never imported locally. This PR adds gossip verification and import when a payload is sent to the http API
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Closes:
- https://github.com/sigp/lighthouse/issues/8689
- Calculate the proposer index on the canonical chain (from canonical head) at `slot` and plumb it through to fork choice so it can be used to determine whether or not to apply the proposer boost. We use the proposer cache to handle state advances and avoid duplicate work.
- Update our FC tests to use `block.message().proposer_index()` (always pass), we are not attempting to test this feature in those tests. The EF tests use the correct canonical proposer idnex via `on_block`, except for invalid blocks which just auto-pass this check (these blocks get rejected by other checks in `on_block` anyway).
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Just a little naming cleanup (no semantic changes) to remove mentions of pending and full states that were still lurking.
This hopefully helps Claude forget about the concept (it defaults to naming variables `pending_state`s without this change).
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
This reverts commit 2749e18d0e, from:
- #9092
We no longer need those changes since the abolition of pending/full states.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
N/A
Adds lints for rust 1.95. Mostly cosmetic.
1. .zip(a.into_iter()) -> .zip(a) . Also a few more places where into_iter is not required
2. replace sort_by with sort_by_key
3. move if statements inside match block.
4. use checked_div instead of if statements. I think this is debatable in terms of being better, happy to remove it if others also feel its unnecessary
Co-Authored-By: Pawan Dhananjay <pawandhananjay@gmail.com>
#9077
Where possible replaces all instances of `validator_monitor::timestamp_now` with `chain.slot_clock.now_duration().unwrap_or_default()`.
Where chain/slot_clock is not available, instead replace it with a convenience function `slot_clock::timestamp_now`.
Remove the `validator_monitor::timestamp_now` function.
Co-Authored-By: Mac L <mjladson@pm.me>
Implement boilerplate for new SSE events as specified in
- https://github.com/ethereum/beacon-APIs/pull/588
While that one is not merged yet, I believe the SSE events might be utilized in Dora already.
Implement the boilerplate, i.e. subscription tracking and publish queues. A PR to implement to fully implement already implementable events will follow.
Co-Authored-By: Daniel Knopik <daniel@dknopik.de>
Closes:
- https://github.com/sigp/lighthouse/issues/9002
- Commit `config.yaml` for minimal and mainnet to `consensus/types/configs`. For now we omit any auto-downloading logic, to avoid the hassles of dealing with Github rate limits etc on CI. Unfortunately these files are NOT bundled inside the spec tests.
- Fix the values of `min_builder_withdrawability_delay` for minimal and mainnet. These discrepancies aren't caught by the current spec tests, because the spec tests are missing data: https://github.com/ethereum/consensus-specs/pull/5005. Will be fixed in the next release/when we update to nightly.
- Fix the blob schedule for `minimal`, which should be empty, NOT inherited from mainnet.
- Keep `SECONDS_PER_SLOT` for now because the Kurtosis tests fail upon their complete removal. We will be able to completely remove `SECONDS_PER_SLOT` soon.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Some of our custom `lighthouse/analysis` endpoints will require maintenance for the Gloas hard fork. We have decided instead to remove those endpoints. We don't utilize them internally and they have pretty limited utility and so we feel they are not worth maintaining.
Remove `lighthouse/analysis/attestation_performance` and `lighthouse/analysis/block_packing_efficiency` endpoints.
Co-Authored-By: Mac L <mjladson@pm.me>
I believe one of our rewards endpoints is slightly out of spec. We do not return the `finalized` status for `post_beacon_rewards_attestations`.
Additionally, the `eth2` client doesn't expect the correct wrapper types for some other endpoints.
- Update `post_beacon_rewards_attestations` server implementation to match spec.
- Update all three client functions in `eth2` to the correct wrapper type.
- Add missing tests for `http_api` to detect any regressions.
Co-Authored-By: Mac L <mjladson@pm.me>
Closes:
- https://github.com/sigp/lighthouse/issues/8869
- Update `BlockReplayer` to support replay of execution payload envelopes.
- Update `HotColdDB` to load payload envelopes and feed them to the `BlockReplayer` for both hot + cold states. However the cold DB code is not fully working yet (see: https://github.com/sigp/lighthouse/issues/8958).
- Add `StatePayloadStatus` to allow callers to specify whether they want a state with a payload applied, or not.
- Fix the state cache to key by `StatePayloadStatus`.
- Lots of fixes to block production and block processing regarding state management.
- Initial test harness support for producing+processing Gloas blocks+envelopes
- A few new tests to cover Gloas DB operations
Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com>
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
