Implement boilerplate for new SSE events as specified in
- https://github.com/ethereum/beacon-APIs/pull/588
While that one is not merged yet, I believe the SSE events might be utilized in Dora already.
Implement the boilerplate, i.e. subscription tracking and publish queues. A PR to implement to fully implement already implementable events will follow.
Co-Authored-By: Daniel Knopik <daniel@dknopik.de>
Closes:
- https://github.com/sigp/lighthouse/issues/9002
- Commit `config.yaml` for minimal and mainnet to `consensus/types/configs`. For now we omit any auto-downloading logic, to avoid the hassles of dealing with Github rate limits etc on CI. Unfortunately these files are NOT bundled inside the spec tests.
- Fix the values of `min_builder_withdrawability_delay` for minimal and mainnet. These discrepancies aren't caught by the current spec tests, because the spec tests are missing data: https://github.com/ethereum/consensus-specs/pull/5005. Will be fixed in the next release/when we update to nightly.
- Fix the blob schedule for `minimal`, which should be empty, NOT inherited from mainnet.
- Keep `SECONDS_PER_SLOT` for now because the Kurtosis tests fail upon their complete removal. We will be able to completely remove `SECONDS_PER_SLOT` soon.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
`serde_yaml` is now deprecated. The API-compatible `yaml_serde` should be used instead.
Replace `serde_yaml` with `yaml_serde`. This is purely mechanical as the API is 1-to-1.
Co-Authored-By: Mac L <mjladson@pm.me>
Some of our custom `lighthouse/analysis` endpoints will require maintenance for the Gloas hard fork. We have decided instead to remove those endpoints. We don't utilize them internally and they have pretty limited utility and so we feel they are not worth maintaining.
Remove `lighthouse/analysis/attestation_performance` and `lighthouse/analysis/block_packing_efficiency` endpoints.
Co-Authored-By: Mac L <mjladson@pm.me>
I believe one of our rewards endpoints is slightly out of spec. We do not return the `finalized` status for `post_beacon_rewards_attestations`.
Additionally, the `eth2` client doesn't expect the correct wrapper types for some other endpoints.
- Update `post_beacon_rewards_attestations` server implementation to match spec.
- Update all three client functions in `eth2` to the correct wrapper type.
- Add missing tests for `http_api` to detect any regressions.
Co-Authored-By: Mac L <mjladson@pm.me>
Fix the issue with the `proposer_duties` endpoint using the wrong dependent root post-Fulu by implementing the new v2 endpoint:
- https://github.com/ethereum/beacon-APIs/pull/563
We need to add this in time for Gloas, and then we can we can deprecate and remove v1.
- Add a new API handler for the v2 endpoint
- Add client code in the `eth2` crate
- Update existing tests and add some new ones to confirm the different behaviour of v1 and v2
There's a bit of test duplication with v1, but this will be resolved once v1 and its tests are deleted.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Co-Authored-By: chonghe <44791194+chong-he@users.noreply.github.com>
Mark pointed out that these APIs will require updates for Gloas, so I figured we may as well get rid of them. As far as I know, blockprint was the only use case and it is now defunct. The consensus block value is included in getBlock API responses, so there's no reason for VCs to use the `POST` API, and there is now a standard API for the rewards of canonical blocks. The SSE event was non-standard, and likely only used by blockprint as well.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
We have duplicated features which enable `arbitrary` throughout the codebase. These are `arbitrary` and `arbitrary-fuzz`. I think historically these were supposed to be distinct however in practice these function identically and so we can unify them into a single feature to avoid confusion.
Co-Authored-By: Mac L <mjladson@pm.me>
Fix a bug in v8.1.0 whereby the VC times out continuously with:
> Feb 18 02:03:48.030 WARN Head service failed retrying starting next slot error: "Head monitoring stream error, node: 0, error: SseClient(Transport(reqwest::Error { kind: Decode, source: reqwest::Error { kind: Body, source: TimedOut } }))"
- Remove the existing timeout for the events API by using `Duration::MAX`. This is necessary as the client is configured with a default timeout. This is the only way to override/remove it.
- DO NOT add a `read_timeout` (yet), as this would need to be configured on a per-client basis. We do not want to create a new Client for every call as the early commits on this branch were doing, as this would bypass the TLS cert config, and is also wasteful.
Co-Authored-By: hopinheimer <knmanas6@gmail.com>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
The flow for local block building is
1. Create execution payload and bid
2. Construct beacon block
3. Sign beacon block and publish
4. Sign execution payload and publish
This PR adds the beacon block v4 flow , GET payload envelope and POST payload envelope (local block building only). The spec for these endpoints can be found here: https://github.com/ethereum/beacon-APIs/pull/552 and is subject to change.
We needed a way to store the unsigned execution payload envelope associated to the execution payload bid that was included in the block. I introduced a new cache that stores these unsigned execution payload envelopes. the GET payload envelope queries this cache directly so that a proposer, after publishing a block, can fetch the payload envelope + sign and publish it.
I kept payload signing and publishing within the validators block service to keep things simple for now. The idea was to build out a block production MVP for devnet 0, try not to affect any non gloas code paths and build things out in such a way that it will be easy to deprecate pre-gloas code paths later on (for example block production v2 and v3).
We will eventually need to track which beacon node was queried for the block so that we can later query it for the payload. But thats not needed for the devnet.
Co-Authored-By: Eitan Seri- Levi <eserilev@gmail.com>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Co-Authored-By: Eitan Seri-Levi <eserilev@ucsc.edu>
`eth2` is currently unable to be built without the `events` feature.
Feature gates the `SseEventSource` match arm in the `status` function.
Co-Authored-By: Mac L <mjladson@pm.me>
I believe I found a bug where during computation of `pid_process_seconds_total` we add `children_system` twice.
I assume that it was originally intended to add `children_system` and `children_user` once each
Co-Authored-By: Mac L <mjladson@pm.me>
This reverts some of the changes from #8524 by adding back the typed network endpoints with an optional `network` feature. Without the `network` feature, these endpoints (and associated dependencies) will not be built.
This means the `enr`, `multiaddr` and `libp2p-identity` dependencies have returned but are now optional
Co-Authored-By: Mac L <mjladson@pm.me>
Pulling out consensus type changes from #8677.
This PR covers all type changes for spec 1.7.0-alpha.1 (except for `DataColumnSidecar` changes, which is covered in @eserilev's PR #8682)
Co-Authored-By: Jimmy Chen <jchen.tc@gmail.com>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
[Missing values in /eth/v1/config/spec #8571
](https://github.com/sigp/lighthouse/issues/8571) - there will be follow up PR for the re org props
1. As per above issue from EF dev ops, I added
```
"EPOCHS_PER_SUBNET_SUBSCRIPTION": "256",
"ATTESTATION_SUBNET_COUNT": "64",
"ATTESTATION_SUBNET_EXTRA_BITS": "0",
"UPDATE_TIMEOUT": "8192",
"DOMAIN_BLS_TO_EXECUTION_CHANGE": "0x0a000000"
```
to `/eth/v1/config/spec`
2. Had to change the minimal config for UPDATE_TIMEOUT to get currents tests to pass. This is ok given UPDATE_TIMEOUT is not used in lighthouse as this config for light client spec from altair
3. ATTESTATION_SUBNET_PREFIX_BITS is now dynamically calculated and shimmed into the /eth/v1/config/spec output as advised by @michaelsproul
Co-Authored-By: Joseph Patchen <josephmipatchen@gmail.com>
Which issue # does this PR address?
#8586
Please list or describe the changes introduced by this PR.
Remove `service_name` from `TaskExecutor`
Co-Authored-By: Abhivansh <31abhivanshj@gmail.com>
#8547
We are currently using an older version of `syn` in `test_random_derive`. Updating this removes one of the sources of `syn` `1.0.109` in our dependency tree.
Co-Authored-By: Mac L <mjladson@pm.me>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
There are certain crates which we re-export within `types` which creates a fragmented DevEx, where there are various ways to import the same crates.
```rust
// consensus/types/src/lib.rs
pub use bls::{
AggregatePublicKey, AggregateSignature, Error as BlsError, Keypair, PUBLIC_KEY_BYTES_LEN,
PublicKey, PublicKeyBytes, SIGNATURE_BYTES_LEN, SecretKey, Signature, SignatureBytes,
get_withdrawal_credentials,
};
pub use context_deserialize::{ContextDeserialize, context_deserialize};
pub use fixed_bytes::FixedBytesExtended;
pub use milhouse::{self, List, Vector};
pub use ssz_types::{BitList, BitVector, FixedVector, VariableList, typenum, typenum::Unsigned};
pub use superstruct::superstruct;
```
This PR removes these re-exports and makes it explicit that these types are imported from a non-`consensus/types` crate.
Co-Authored-By: Mac L <mjladson@pm.me>
Previously, we had a pinned version of `alloy` to fix some crate compatibility issues we encountered during the migration away from `ethers`. Now that the migration is complete we should remove the pin. This also updates alloy crates to their latest versions.
Co-Authored-By: Mac L <mjladson@pm.me>
Remove certain dependencies from `eth2`, and feature-gate others which are only used by certain endpoints.
| Removed | Optional | Dev only |
| -------- | -------- | -------- |
| `either` `enr` `libp2p-identity` `multiaddr` | `protoarray` `eth2_keystore` `eip_3076` `zeroize` `reqwest-eventsource` `futures` `futures-util` | `rand` `test_random_derive` |
This is done by adding an `events` feature which enables the events endpoint and its associated dependencies.
The `lighthouse` feature also enables its associated dependencies making them optional.
The networking-adjacent dependencies were removed by just having certain fields use a `String` instead of an explicit network type. This means the user should handle conversion at the call site instead. This is a bit spicy, but I believe `PeerId`, `Enr` and `Multiaddr` are easily converted to and from `String`s so I think it's fine and reduces our dependency space by a lot. The alternative is to feature gate these types behind a `network` feature instead.
Co-Authored-By: Mac L <mjladson@pm.me>
Organize and categorize `consensus/types` into modules based on their relation to key consensus structures/concepts.
This is a precursor to a sensible public interface.
While this refactor is very opinionated, I am open to suggestions on module names, or type groupings if my current ones are inappropriate.
Co-Authored-By: Mac L <mjladson@pm.me>
This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the `Makefile` to streamline reproducible build configurations, and modifies the `Dockerfile.reproducible` to align with the new build process. Additionally, it removes the `reproducible` profile from `Cargo.toml`.
### New GitHub Actions Workflows:
* [`.github/workflows/docker-reproducible.yml`](diffhunk://#diff-222af23bee616920b04f5b92a83eb5106fce08abd885cd3a3b15b8beb5e789c3R1-R145): Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs without pushing an image.
### Build Configuration Updates:
* [`Makefile`](diffhunk://#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52L85-R143): Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments for `x86_64` and `aarch64` architectures.
* [`Dockerfile.reproducible`](diffhunk://#diff-587298ff141278ce3be7c54a559f9f31472cc5b384e285e2105b3dee319ba31dL1-R24): Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to the `Makefile`.
* Switch to using jemalloc-sys from Debian repos instead of building it from source. A Debian version is [reproducible](https://tests.reproducible-builds.org/debian/rb-pkg/trixie/amd64/jemalloc.html) which is [hard to achieve](https://github.com/NixOS/nixpkgs/issues/380852) if you build it from source.
### Profile Removal:
* [`Cargo.toml`](diffhunk://#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L289-L295): Removes the `reproducible` profile, simplifying build configurations and relying on external tooling for reproducibility.
Co-Authored-By: Moe Mahhouk <mohammed-mahhouk@hotmail.com>
Co-Authored-By: chonghe <44791194+chong-he@users.noreply.github.com>
Co-Authored-By: Michael Sproul <michaelsproul@users.noreply.github.com>
Fixes#7785
- [x] Update all integration tests with >1 files to follow the `main` pattern.
- [x] `crypto/eth2_key_derivation/tests`
- [x] `crypto/eth2_keystore/tests`
- [x] `crypto/eth2_wallet/tests`
- [x] `slasher/tests`
- [x] `common/eth2_interop_keypairs/tests`
- [x] `beacon_node/lighthouse_network/tests`
- [x] Set `debug_assertions` to false on `.vscode/settings.json`.
- [x] Document how to make rust analyzer work on integration tests files. In `book/src/contributing_setup.md`
---
Tracking a `rust-analyzer.toml` with settings like the one provided in `.vscode/settings.json` would be nicer. But this is not possible yet. For now, that config should be a good enough indicator for devs using editors different to VSCode.
Co-Authored-By: Daniel Ramirez-Chiquillo <hi@danielrachi.com>
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
Update `reqwest` to 0.12 so we only depend on a single version. This should slightly improve compile times and reduce binary bloat.
Co-Authored-By: Michael Sproul <michael@sigmaprime.io>
