gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-03-21 22:04:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

Improving blob propagation post-PeerDAS with Decentralized Blob Building (#6268)

Browse Source 
* Get blobs from EL.

Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Avoid cloning blobs after fetching blobs.

* Address review comments and refactor code.

* Fix lint.

* Move blob computation metric to the right spot.

* Merge branch 'unstable' into das-fetch-blobs

* Merge branch 'unstable' into das-fetch-blobs

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs
#	beacon_node/beacon_chain/src/block_verification.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs

* Merge branch 'unstable' into das-fetch-blobs

# Conflicts:
#	beacon_node/beacon_chain/src/beacon_chain.rs

* Gradual publication of data columns for supernodes.

* Recompute head after importing block with blobs from the EL.

* Fix lint

* Merge branch 'unstable' into das-fetch-blobs

* Use blocking task instead of async when computing cells.

* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs

* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs

* Fix semantic conflicts

* Downgrade error log.

* Merge branch 'unstable' into das-fetch-blobs

# Conflicts:
#	beacon_node/beacon_chain/src/data_availability_checker.rs
#	beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs
#	beacon_node/execution_layer/src/engine_api.rs
#	beacon_node/execution_layer/src/engine_api/json_structures.rs
#	beacon_node/network/src/network_beacon_processor/gossip_methods.rs
#	beacon_node/network/src/network_beacon_processor/mod.rs
#	beacon_node/network/src/network_beacon_processor/sync_methods.rs

* Merge branch 'unstable' into das-fetch-blobs

* Publish block without waiting for blob and column proof computation.

* Address review comments and refactor.

* Merge branch 'unstable' into das-fetch-blobs

* Fix test and docs.

* Comment cleanups.

* Merge branch 'unstable' into das-fetch-blobs

* Address review comments and cleanup

* Address review comments and cleanup

* Refactor to de-duplicate gradual publication logic.

* Add more logging.

* Merge remote-tracking branch 'origin/unstable' into das-fetch-blobs

# Conflicts:
#	Cargo.lock

* Fix incorrect comparison on `num_fetched_blobs`.

* Implement gradual blob publication.

* Merge branch 'unstable' into das-fetch-blobs

* Inline `publish_fn`.

* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs

* Gossip verify blobs before publishing

* Avoid queries for 0 blobs and error for duplicates

* Gossip verified engine blob before processing them, and use observe cache to detect duplicates before publishing.

* Merge branch 'das-fetch-blobs' of github.com:jimmygchen/lighthouse into das-fetch-blobs

# Conflicts:
#	beacon_node/network/src/network_beacon_processor/mod.rs

* Merge branch 'unstable' into das-fetch-blobs

* Fix invalid commitment inclusion proofs in blob sidecars created from EL blobs.

* Only publish EL blobs triggered from gossip block, and not RPC block.

* Downgrade gossip blob log to `debug`.

* Merge branch 'unstable' into das-fetch-blobs

* Merge branch 'unstable' into das-fetch-blobs

* Grammar
This commit is contained in:
Jimmy Chen
2024-11-15 10:34:13 +07:00
committed by GitHub
parent 8e95024945
commit 5f053b0b6d
36 changed files with 1660 additions and 613 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
consensus/types/src/beacon_block_body.rs
View File

@@ -147,7 +147,7 @@ impl<'a, E: EthSpec, Payload: AbstractExecPayload<E>> BeaconBlockBodyRef<'a, E,
}
}
fn body_merkle_leaves(&self) -> Vec<Hash256> {
pub(crate) fn body_merkle_leaves(&self) -> Vec<Hash256> {
let mut leaves = vec![];
match self {
Self::Base(body) => {
@@ -178,57 +178,71 @@ impl<'a, E: EthSpec, Payload: AbstractExecPayload<E>> BeaconBlockBodyRef<'a, E,
leaves
}
/// Produces the proof of inclusion for a `KzgCommitment` in `self.blob_kzg_commitments`
/// at `index`.
/// Calculate a KZG commitment merkle proof.
///
/// Prefer to use `complete_kzg_commitment_merkle_proof` with a reused proof for the
/// `blob_kzg_commitments` field.
pub fn kzg_commitment_merkle_proof(
&self,
index: usize,
) -> Result<FixedVector<Hash256, E::KzgCommitmentInclusionProofDepth>, Error> {
// We compute the branches by generating 2 merkle trees:
// 1. Merkle tree for the `blob_kzg_commitments` List object
// 2. Merkle tree for the `BeaconBlockBody` container
// We then merge the branches for both the trees all the way up to the root.
let kzg_commitments_proof = self.kzg_commitments_merkle_proof()?;
let proof = self.complete_kzg_commitment_merkle_proof(index, &kzg_commitments_proof)?;
Ok(proof)
}
// Part1 (Branches for the subtree rooted at `blob_kzg_commitments`)
//
// Branches for `blob_kzg_commitments` without length mix-in
let blob_leaves = self
.blob_kzg_commitments()?
.iter()
.map(|commitment| commitment.tree_hash_root())
.collect::<Vec<_>>();
let depth = E::max_blob_commitments_per_block()
.next_power_of_two()
.ilog2();
let tree = MerkleTree::create(&blob_leaves, depth as usize);
let (_, mut proof) = tree
.generate_proof(index, depth as usize)
.map_err(Error::MerkleTreeError)?;
/// Produces the proof of inclusion for a `KzgCommitment` in `self.blob_kzg_commitments`
/// at `index` using an existing proof for the `blob_kzg_commitments` field.
pub fn complete_kzg_commitment_merkle_proof(
&self,
index: usize,
kzg_commitments_proof: &[Hash256],
) -> Result<FixedVector<Hash256, E::KzgCommitmentInclusionProofDepth>, Error> {
match self {
Self::Base(_) | Self::Altair(_) | Self::Bellatrix(_) | Self::Capella(_) => {
Err(Error::IncorrectStateVariant)
}
Self::Deneb(_) | Self::Electra(_) => {
// We compute the branches by generating 2 merkle trees:
// 1. Merkle tree for the `blob_kzg_commitments` List object
// 2. Merkle tree for the `BeaconBlockBody` container
// We then merge the branches for both the trees all the way up to the root.
// Add the branch corresponding to the length mix-in.
let length = blob_leaves.len();
let usize_len = std::mem::size_of::<usize>();
let mut length_bytes = [0; BYTES_PER_CHUNK];
length_bytes
.get_mut(0..usize_len)
.ok_or(Error::MerkleTreeError(MerkleTreeError::PleaseNotifyTheDevs))?
.copy_from_slice(&length.to_le_bytes());
let length_root = Hash256::from_slice(length_bytes.as_slice());
proof.push(length_root);
// Part1 (Branches for the subtree rooted at `blob_kzg_commitments`)
//
// Branches for `blob_kzg_commitments` without length mix-in
let blob_leaves = self
.blob_kzg_commitments()?
.iter()
.map(|commitment| commitment.tree_hash_root())
.collect::<Vec<_>>();
let depth = E::max_blob_commitments_per_block()
.next_power_of_two()
.ilog2();
let tree = MerkleTree::create(&blob_leaves, depth as usize);
let (_, mut proof) = tree
.generate_proof(index, depth as usize)
.map_err(Error::MerkleTreeError)?;
// Part 2
// Branches for `BeaconBlockBody` container
let body_leaves = self.body_merkle_leaves();
let beacon_block_body_depth = body_leaves.len().next_power_of_two().ilog2() as usize;
let tree = MerkleTree::create(&body_leaves, beacon_block_body_depth);
let (_, mut proof_body) = tree
.generate_proof(BLOB_KZG_COMMITMENTS_INDEX, beacon_block_body_depth)
.map_err(Error::MerkleTreeError)?;
// Join the proofs for the subtree and the main tree
proof.append(&mut proof_body);
debug_assert_eq!(proof.len(), E::kzg_proof_inclusion_proof_depth());
// Add the branch corresponding to the length mix-in.
let length = blob_leaves.len();
let usize_len = std::mem::size_of::<usize>();
let mut length_bytes = [0; BYTES_PER_CHUNK];
length_bytes
.get_mut(0..usize_len)
.ok_or(Error::MerkleTreeError(MerkleTreeError::PleaseNotifyTheDevs))?
.copy_from_slice(&length.to_le_bytes());
let length_root = Hash256::from_slice(length_bytes.as_slice());
proof.push(length_root);
Ok(proof.into())
// Part 2
// Branches for `BeaconBlockBody` container
// Join the proofs for the subtree and the main tree
proof.extend_from_slice(kzg_commitments_proof);
Ok(FixedVector::new(proof)?)
}
}
}
/// Produces the proof of inclusion for `self.blob_kzg_commitments`.
@@ -241,7 +255,7 @@ impl<'a, E: EthSpec, Payload: AbstractExecPayload<E>> BeaconBlockBodyRef<'a, E,
let (_, proof) = tree
.generate_proof(BLOB_KZG_COMMITMENTS_INDEX, beacon_block_body_depth)
.map_err(Error::MerkleTreeError)?;
Ok(proof.into())
Ok(FixedVector::new(proof)?)
}
pub fn block_body_merkle_proof(&self, generalized_index: usize) -> Result<Vec<Hash256>, Error> {

31
consensus/types/src/blob_sidecar.rs
View File

@@ -150,6 +150,37 @@ impl<E: EthSpec> BlobSidecar<E> {
})
}
pub fn new_with_existing_proof(
index: usize,
blob: Blob<E>,
signed_block: &SignedBeaconBlock<E>,
signed_block_header: SignedBeaconBlockHeader,
kzg_commitments_inclusion_proof: &[Hash256],
kzg_proof: KzgProof,
) -> Result<Self, BlobSidecarError> {
let expected_kzg_commitments = signed_block
.message()
.body()
.blob_kzg_commitments()
.map_err(|_e| BlobSidecarError::PreDeneb)?;
let kzg_commitment = *expected_kzg_commitments
.get(index)
.ok_or(BlobSidecarError::MissingKzgCommitment)?;
let kzg_commitment_inclusion_proof = signed_block
.message()
.body()
.complete_kzg_commitment_merkle_proof(index, kzg_commitments_inclusion_proof)?;
Ok(Self {
index: index as u64,
blob,
kzg_commitment,
kzg_proof,
signed_block_header,
kzg_commitment_inclusion_proof,
})
}
pub fn id(&self) -> BlobIdentifier {
BlobIdentifier {
block_root: self.block_root(),

42
consensus/types/src/signed_beacon_block.rs
View File

@@ -1,6 +1,7 @@
use crate::beacon_block_body::format_kzg_commitments;
use crate::beacon_block_body::{format_kzg_commitments, BLOB_KZG_COMMITMENTS_INDEX};
use crate::*;
use derivative::Derivative;
use merkle_proof::MerkleTree;
use serde::{Deserialize, Serialize};
use ssz_derive::{Decode, Encode};
use std::fmt;
@@ -239,6 +240,45 @@ impl<E: EthSpec, Payload: AbstractExecPayload<E>> SignedBeaconBlock<E, Payload>
}
}
/// Produce a signed beacon block header AND a merkle proof for the KZG commitments.
///
/// This method is more efficient than generating each part separately as it reuses hashing.
pub fn signed_block_header_and_kzg_commitments_proof(
&self,
) -> Result<
(
SignedBeaconBlockHeader,
FixedVector<Hash256, E::KzgCommitmentsInclusionProofDepth>,
),
Error,
> {
// Create the block body merkle tree
let body_leaves = self.message().body().body_merkle_leaves();
let beacon_block_body_depth = body_leaves.len().next_power_of_two().ilog2() as usize;
let body_merkle_tree = MerkleTree::create(&body_leaves, beacon_block_body_depth);
// Compute the KZG commitments inclusion proof
let (_, proof) = body_merkle_tree
.generate_proof(BLOB_KZG_COMMITMENTS_INDEX, beacon_block_body_depth)
.map_err(Error::MerkleTreeError)?;
let kzg_commitments_inclusion_proof = FixedVector::new(proof)?;
let block_header = BeaconBlockHeader {
slot: self.slot(),
proposer_index: self.message().proposer_index(),
parent_root: self.parent_root(),
state_root: self.state_root(),
body_root: body_merkle_tree.hash(),
};
let signed_header = SignedBeaconBlockHeader {
message: block_header,
signature: self.signature().clone(),
};
Ok((signed_header, kzg_commitments_inclusion_proof))
}
/// Convenience accessor for the block's slot.
pub fn slot(&self) -> Slot {
self.message().slot()

29
consensus/types/src/test_utils/generate_random_block_and_blobs.rs
View File

@@ -83,6 +83,35 @@ mod test {
}
}
#[test]
fn test_verify_blob_inclusion_proof_from_existing_proof() {
let (block, mut blob_sidecars) =
generate_rand_block_and_blobs::<MainnetEthSpec>(ForkName::Deneb, 1, &mut thread_rng());
let BlobSidecar {
index,
blob,
kzg_proof,
..
} = blob_sidecars.pop().unwrap();
// Compute the commitments inclusion proof and use it for building blob sidecar.
let (signed_block_header, kzg_commitments_inclusion_proof) = block
.signed_block_header_and_kzg_commitments_proof()
.unwrap();
let blob_sidecar = BlobSidecar::new_with_existing_proof(
index as usize,
blob,
&block,
signed_block_header,
&kzg_commitments_inclusion_proof,
kzg_proof,
)
.unwrap();
assert!(blob_sidecar.verify_blob_sidecar_inclusion_proof());
}
#[test]
fn test_verify_blob_inclusion_proof_invalid() {
let (_block, blobs) =