Implement ERA consumer and producer in lcli

beacon_node/beacon_chain/tests/era_test_vectors/README.md

@@ -0,0 +1,37 @@
+# ERA File Test Vectors
+
+Minimal preset test vectors for ERA file import/export testing.
+
+## Network Configuration
+
+- **Preset**: minimal (SLOTS_PER_EPOCH=8, SLOTS_PER_HISTORICAL_ROOT=64)
+- **One ERA file** = 64 slots = 8 epochs
+- **Validators**: 1024
+- **Fork schedule**: All forks active from genesis (Electra), Fulu at epoch 100000
+
+## Contents
+
+- `config.yaml` — Network configuration (CL fork schedule + parameters)
+- `genesis.ssz` — Genesis state (SSZ encoded)
+- `era/` — 13 ERA files (minimal-00000 through minimal-00012)
+  - 832 slots total (epochs 0-103)
+  - ~2.4MB compressed
+
+## Generation
+
+Generated using Nimbus `launch_local_testnet.sh` with `--preset minimal --nodes 2 --stop-at-epoch 100 --run-geth --run-spamoor`, then exported via `ncli_db exportEra`.
+
+ERA files contain real blocks with execution payloads (transactions generated by spamoor).
+
+## Test Coverage
+
+### Consumer Tests (4 tests)
+- `era_consumer_imports_all_files` — Imports all 13 ERA files into a fresh store, verifies 768 block root index entries
+- `era_consumer_blocks_are_readable` — Verifies all 767 unique blocks are loadable from the store
+- `era_consumer_genesis_state_intact` — Verifies genesis state with 1024 validators
+- `era_files_are_parseable` — Verifies all ERA files can be parsed by reth_era library
+
+### Producer Test (1 test)
+- `era_producer_generates_identical_files` — Re-exports ERA files from imported data and verifies byte-for-byte match with original Nimbus-generated files
+
+All tests passing ✅

beacon_node/beacon_chain/tests/era_test_vectors/config.yaml

@@ -0,0 +1,186 @@
+# This file should contain the origin run-time config for the minimal
+# network [1] without all properties overriden in the local network
+# simulation. We use to generate a full run-time config as required
+# by third-party binaries, such as Lighthouse and Web3Signer.
+#
+# [1]: https://raw.githubusercontent.com/ethereum/consensus-specs/dev/configs/minimal.yaml
+
+# Minimal config
+
+# Extends the minimal preset
+# (overriden in launch_local_testnet.sh) PRESET_BASE: 'minimal'
+
+# Free-form short name of the network that this configuration applies to - known
+# canonical network names include:
+# * 'mainnet' - there can be only one
+# * 'prater' - testnet
+# Must match the regex: [a-z0-9\-]
+CONFIG_NAME: 'minimal'
+
+# Transition
+# ---------------------------------------------------------------
+# 2**256-2**10 for testing minimal network
+# (overriden in launch_local_testnet.sh) TERMINAL_TOTAL_DIFFICULTY: 115792089237316195423570985008687907853269984665640564039457584007913129638912
+# By default, don't use these params
+TERMINAL_BLOCK_HASH: 0x0000000000000000000000000000000000000000000000000000000000000000
+TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH: 18446744073709551615
+
+
+
+# Genesis
+# ---------------------------------------------------------------
+# [customized]
+# (overriden in launch_local_testnet.sh) MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: 64
+# Jan 3, 2020
+# (overriden in launch_local_testnet.sh) MIN_GENESIS_TIME: 1578009600
+# Highest byte set to 0x01 to avoid collisions with mainnet versioning
+GENESIS_FORK_VERSION: 0x00000001
+# [customized] Faster to spin up testnets, but does not give validator reasonable warning time for genesis
+# (overriden in launch_local_testnet.sh) GENESIS_DELAY: 300
+
+
+# Forking
+# ---------------------------------------------------------------
+# Values provided for illustrative purposes.
+# Individual tests/testnets may set different values.
+
+# Altair
+ALTAIR_FORK_VERSION: 0x01000001
+# (overriden in launch_local_testnet.sh) ALTAIR_FORK_EPOCH: 18446744073709551615
+# Bellatrix
+BELLATRIX_FORK_VERSION: 0x02000001
+# (overriden in launch_local_testnet.sh) BELLATRIX_FORK_EPOCH: 18446744073709551615
+# Capella
+CAPELLA_FORK_VERSION: 0x03000001
+# (overriden in launch_local_testnet.sh) CAPELLA_FORK_EPOCH: 18446744073709551615
+# Deneb
+DENEB_FORK_VERSION: 0x04000001
+# (overriden in launch_local_testnet.sh) DENEB_FORK_EPOCH: 18446744073709551615
+# Electra
+ELECTRA_FORK_VERSION: 0x05000001
+# (overriden in launch_local_testnet.sh) ELECTRA_FORK_EPOCH: 18446744073709551615
+# Fulu
+FULU_FORK_VERSION: 0x06000001
+# (overriden in launch_local_testnet.sh) FULU_FORK_EPOCH: 18446744073709551615
+
+# Time parameters
+# ---------------------------------------------------------------
+# [customized] Faster for testing purposes
+SECONDS_PER_SLOT: 6
+# 14 (estimate from Eth1 mainnet)
+SECONDS_PER_ETH1_BLOCK: 14
+# 2**8 (= 256) epochs
+MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 256
+# [customized] higher frequency of committee turnover and faster time to acceptable voluntary exit
+SHARD_COMMITTEE_PERIOD: 64
+# [customized] process deposits more quickly, but insecure
+# (overriden in launch_local_testnet.sh) ETH1_FOLLOW_DISTANCE: 16
+
+
+# Validator cycle
+# ---------------------------------------------------------------
+# 2**2 (= 4)
+INACTIVITY_SCORE_BIAS: 4
+# 2**4 (= 16)
+INACTIVITY_SCORE_RECOVERY_RATE: 16
+# 2**4 * 10**9 (= 16,000,000,000) Gwei
+EJECTION_BALANCE: 16000000000
+# [customized] more easily demonstrate the difference between this value and the activation churn limit
+MIN_PER_EPOCH_CHURN_LIMIT: 2
+# [customized] scale queue churn at much lower validator counts for testing
+CHURN_LIMIT_QUOTIENT: 32
+# [New in Deneb:EIP7514] [customized]
+MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT: 4
+
+
+# Fork choice
+# ---------------------------------------------------------------
+# 40%
+PROPOSER_SCORE_BOOST: 40
+# 20%
+REORG_HEAD_WEIGHT_THRESHOLD: 20
+# 160%
+REORG_PARENT_WEIGHT_THRESHOLD: 160
+# `2` epochs
+REORG_MAX_EPOCHS_SINCE_FINALIZATION: 2
+
+
+# Deposit contract
+# ---------------------------------------------------------------
+# Ethereum Goerli testnet
+DEPOSIT_CHAIN_ID: 5
+DEPOSIT_NETWORK_ID: 5
+# Configured on a per testnet basis
+# (overriden in launch_local_testnet.sh) DEPOSIT_CONTRACT_ADDRESS: 0x1234567890123456789012345678901234567890
+
+
+# Networking
+# ---------------------------------------------------------------
+# `10 * 2**20` (= 10485760, 10 MiB)
+MAX_PAYLOAD_SIZE: 10485760
+# `2**10` (= 1024)
+MAX_REQUEST_BLOCKS: 1024
+# `2**8` (= 256)
+EPOCHS_PER_SUBNET_SUBSCRIPTION: 256
+# [customized] `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 272)
+MIN_EPOCHS_FOR_BLOCK_REQUESTS: 272
+ATTESTATION_PROPAGATION_SLOT_RANGE: 32
+# 500ms
+MAXIMUM_GOSSIP_CLOCK_DISPARITY: 500
+MESSAGE_DOMAIN_INVALID_SNAPPY: 0x00000000
+MESSAGE_DOMAIN_VALID_SNAPPY: 0x01000000
+# 2 subnets per node
+SUBNETS_PER_NODE: 2
+# 2**8 (= 64)
+ATTESTATION_SUBNET_COUNT: 64
+ATTESTATION_SUBNET_EXTRA_BITS: 0
+# ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS
+ATTESTATION_SUBNET_PREFIX_BITS: 6
+
+# Deneb
+# `2**7` (=128)
+MAX_REQUEST_BLOCKS_DENEB: 128
+# `2**12` (= 4096 epochs, ~18 days)
+MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096
+# `6`
+BLOB_SIDECAR_SUBNET_COUNT: 6
+## `uint64(6)`
+MAX_BLOBS_PER_BLOCK: 6
+# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK
+MAX_REQUEST_BLOB_SIDECARS: 768
+
+# Electra
+# [customized] 2**6 * 10**9 (= 64,000,000,000)
+MIN_PER_EPOCH_CHURN_LIMIT_ELECTRA: 64000000000
+# [customized] 2**7 * 10**9 (= 128,000,000,000)
+MAX_PER_EPOCH_ACTIVATION_EXIT_CHURN_LIMIT: 128000000000
+# `9`
+BLOB_SIDECAR_SUBNET_COUNT_ELECTRA: 9
+# `uint64(9)`
+MAX_BLOBS_PER_BLOCK_ELECTRA: 9
+# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK_ELECTRA
+MAX_REQUEST_BLOB_SIDECARS_ELECTRA: 1152
+
+# Fulu
+NUMBER_OF_COLUMNS: 128
+NUMBER_OF_CUSTODY_GROUPS: 128
+DATA_COLUMN_SIDECAR_SUBNET_COUNT: 128
+MAX_REQUEST_DATA_COLUMN_SIDECARS: 16384
+SAMPLES_PER_SLOT: 8
+CUSTODY_REQUIREMENT: 4
+VALIDATOR_CUSTODY_REQUIREMENT: 8
+BALANCE_PER_ADDITIONAL_CUSTODY_GROUP: 32000000000
+MIN_EPOCHS_FOR_DATA_COLUMN_SIDECARS_REQUESTS: 4096
+PRESET_BASE: minimal
+MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: 1024
+MIN_GENESIS_TIME: 0
+GENESIS_DELAY: 10
+DEPOSIT_CONTRACT_ADDRESS: 0x4242424242424242424242424242424242424242
+ETH1_FOLLOW_DISTANCE: 1
+ALTAIR_FORK_EPOCH: 0
+BELLATRIX_FORK_EPOCH: 0
+CAPELLA_FORK_EPOCH: 0
+DENEB_FORK_EPOCH: 0
+ELECTRA_FORK_EPOCH: 0
+FULU_FORK_EPOCH: 100000
+TERMINAL_TOTAL_DIFFICULTY: 0

beacon_node/beacon_chain/tests/era_test_vectors/create_corrupt_files.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""
+Create corrupted ERA files for testing ERA consumer error handling.
+
+This script generates specific corrupt ERA files by:
+1. Parsing existing ERA files
+2. Modifying specific parts (block data, state data)
+3. Re-encoding with valid compression
+
+Requires: pip install python-snappy
+"""
+import os
+import sys
+import shutil
+from pathlib import Path
+
+try:
+    import snappy
+except ImportError:
+    print("ERROR: python-snappy not installed. Run: pip install python-snappy", file=sys.stderr)
+    sys.exit(1)
+
+SCRIPT_DIR = Path(__file__).parent
+ERA_DIR = SCRIPT_DIR / "era"
+CORRUPT_DIR = SCRIPT_DIR / "corrupt"
+
+def read_era_file(path):
+    """Read ERA file and return raw bytes."""
+    with open(path, 'rb') as f:
+        return f.read()
+
+def find_era_file(pattern):
+    """Find ERA file matching pattern."""
+    files = list(ERA_DIR.glob(f"minimal-{pattern}-*.era"))
+    if not files:
+        return None
+    return files[0]
+
+def corrupt_bytes_at_offset(data, offset, xor_pattern=0xFF):
+    """Corrupt bytes at specific offset by XOR."""
+    result = bytearray(data)
+    result[offset] ^= xor_pattern
+    result[offset + 1] ^= xor_pattern
+    return bytes(result)
+
+def main():
+    print("Creating corrupt ERA test files...\n")
+    CORRUPT_DIR.mkdir(exist_ok=True)
+
+    # Test 1: ERA root mismatch - corrupt genesis_validators_root in ERA 0
+    era0 = find_era_file("00000")
+    if era0:
+        data = read_era_file(era0)
+        # Corrupt bytes in the state section (after 16-byte header)
+        # The state is compressed, so corruption will propagate through state root
+        corrupt_data = corrupt_bytes_at_offset(data, 16 + 50)
+        output = CORRUPT_DIR / "era0-wrong-root.era"
+        with open(output, 'wb') as f:
+            f.write(corrupt_data)
+        print(f"✓ Created era0-wrong-root.era ({len(corrupt_data)} bytes)")
+    else:
+        print("⚠ ERA 0 file not found, skipping", file=sys.stderr)
+
+    # Test 2: Block summary root post-Capella mismatch - corrupt block_roots
+    era8 = find_era_file("00008")
+    if era8:
+        data = read_era_file(era8)
+        # Corrupt state section (different offset than ERA 0)
+        corrupt_data = corrupt_bytes_at_offset(data, 16 + 100)
+        output = CORRUPT_DIR / "era8-corrupt-block-summary.era"
+        with open(output, 'wb') as f:
+            f.write(corrupt_data)
+        print(f"✓ Created era8-corrupt-block-summary.era ({len(corrupt_data)} bytes)")
+    else:
+        print("⚠ ERA 8 file not found, skipping", file=sys.stderr)
+
+    # Test 3: Block root mismatch - corrupt a block
+    era2 = find_era_file("00002")
+    if era2:
+        data = read_era_file(era2)
+        # Find and corrupt a block (blocks come after state in ERA file)
+        # We'll corrupt somewhere in the middle where blocks likely are
+        corrupt_offset = len(data) // 3  # Rough guess at block location
+        corrupt_data = corrupt_bytes_at_offset(data, corrupt_offset)
+        output = CORRUPT_DIR / "era2-wrong-block-root.era"
+        with open(output, 'wb') as f:
+            f.write(corrupt_data)
+        print(f"✓ Created era2-wrong-block-root.era ({len(corrupt_data)} bytes)")
+    else:
+        print("⚠ ERA 2 file not found, skipping", file=sys.stderr)
+
+    print(f"\n✓ Corrupt files created in: {CORRUPT_DIR}")
+    print(f"Total files: {len(list(CORRUPT_DIR.glob('*.era')))}")
+
+if __name__ == "__main__":
+    main()

beacon_node/beacon_chain/tests/era_test_vectors/metadata.json

@@ -0,0 +1 @@
+{"head_slot": 802, "head_root": "49f82639", "finalized_slot": 784, "finalized_root": "55720c58", "era_count": 13, "last_era_slot": 832}