mirror of
https://github.com/sigp/lighthouse.git
synced 2026-03-06 18:21:45 +00:00
Rate limiting backfill sync (#3936)
## Issue Addressed #3212 ## Proposed Changes - Introduce a new `rate_limiting_backfill_queue` - any new inbound backfill work events gets immediately sent to this FIFO queue **without any processing** - Spawn a `backfill_scheduler` routine that pops a backfill event from the FIFO queue at specified intervals (currently halfway through a slot, or at 6s after slot start for 12s slots) and sends the event to `BeaconProcessor` via a `scheduled_backfill_work_tx` channel - This channel gets polled last in the `InboundEvents`, and work event received is wrapped in a `InboundEvent::ScheduledBackfillWork` enum variant, which gets processed immediately or queued by the `BeaconProcessor` (existing logic applies from here) Diagram comparing backfill processing with / without rate-limiting: https://github.com/sigp/lighthouse/issues/3212#issuecomment-1386249922 See this comment for @paulhauner's explanation and solution: https://github.com/sigp/lighthouse/issues/3212#issuecomment-1384674956 ## Additional Info I've compared this branch (with backfill processing rate limited to to 1 and 3 batches per slot) against the latest stable version. The CPU usage during backfill sync is reduced by ~5% - 20%, more details on this page: https://hackmd.io/@jimmygchen/SJuVpJL3j The above testing is done on Goerli (as I don't currently have hardware for Mainnet), I'm guessing the differences are likely to be bigger on mainnet due to block size. ### TODOs - [x] Experiment with processing multiple batches per slot. (need to think about how to do this for different slot durations) - [x] Add option to disable rate-limiting, enabed by default. - [x] (No longer required now we're reusing the reprocessing queue) Complete the `backfill_scheduler` task when backfill sync is completed or not required
This commit is contained in:
@@ -261,6 +261,14 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
|
||||
.min_values(0)
|
||||
.hidden(true)
|
||||
)
|
||||
.arg(
|
||||
Arg::with_name("disable-backfill-rate-limiting")
|
||||
.long("disable-backfill-rate-limiting")
|
||||
.help("Disable the backfill sync rate-limiting. This allow users to just sync the entire chain as fast \
|
||||
as possible, however it can result in resource contention which degrades staking performance. Stakers \
|
||||
should generally choose to avoid this flag since backfill sync is not required for staking.")
|
||||
.takes_value(false),
|
||||
)
|
||||
/* REST API related arguments */
|
||||
.arg(
|
||||
Arg::with_name("http")
|
||||
|
||||
@@ -759,6 +759,10 @@ pub fn get_config<E: EthSpec>(
|
||||
client_config.always_prefer_builder_payload = true;
|
||||
}
|
||||
|
||||
// Backfill sync rate-limiting
|
||||
client_config.chain.enable_backfill_rate_limiting =
|
||||
!cli_args.is_present("disable-backfill-rate-limiting");
|
||||
|
||||
Ok(client_config)
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user