gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-03-19 04:42:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Pause sync when EE is offline (#3428)

Browse Source 
## Issue Addressed

#3032

## Proposed Changes

Pause sync when ee is offline. Changes include three main parts:
- Online/offline notification system
- Pause sync
- Resume sync

#### Online/offline notification system
- The engine state is now guarded behind a new struct `State` that ensures every change is correctly notified. Notifications are only sent if the state changes. The new `State` is behind a `RwLock` (as before) as the synchronization mechanism.
- The actual notification channel is a [tokio::sync::watch](https://docs.rs/tokio/latest/tokio/sync/watch/index.html) which ensures only the last value is in the receiver channel. This way we don't need to worry about message order etc.
- Sync waits for state changes concurrently with normal messages.

#### Pause Sync
Sync has four components, pausing is done differently in each:
- **Block lookups**: Disabled while in this state. We drop current requests and don't search for new blocks. Block lookups are infrequent and I don't think it's worth the extra logic of keeping these and delaying processing. If we later see that this is required, we can add it.
- **Parent lookups**: Disabled while in this state. We drop current requests and don't search for new parents. Parent lookups are even less frequent and I don't think it's worth the extra logic of keeping these and delaying processing. If we later see that this is required, we can add it.
- **Range**: Chains don't send batches for processing to the beacon processor. This is easily done by guarding the channel to the beacon processor and giving it access only if the ee is responsive. I find this the simplest and most powerful approach since we don't need to deal with new sync states and chain segments that are added while the ee is offline will follow the same logic without needing to synchronize a shared state among those. Another advantage of passive pause vs active pause is that we can still keep track of active advertised chain segments so that on resume we don't need to re-evaluate all our peers.
- **Backfill**: Not affected by ee states, we don't pause.

#### Resume Sync
- **Block lookups**: Enabled again.
- **Parent lookups**: Enabled again.
- **Range**: Active resume. Since the only real pause range does is not sending batches for processing, resume makes all chains that are holding read-for-processing batches send them.
- **Backfill**: Not affected by ee states, no need to resume.

## Additional Info

**QUESTION**: Originally I made this to notify and change on synced state, but @pawanjay176 on talks with @paulhauner concluded we only need to check online/offline states. The upcheck function mentions extra checks to have a very up to date sync status to aid the networking stack. However, the only need the networking stack would have is this one. I added a TODO to review if the extra check can be removed

Next gen of #3094

Will work best with #3439 

Co-authored-by: Pawan Dhananjay <pawandhananjay@gmail.com>
This commit is contained in:
Divma
2022-08-24 23:34:56 +00:00
parent aab4a8d2f2
commit 8c69d57c2c
14 changed files with 574 additions and 328 deletions
Download Patch File Download Diff File Expand all files Collapse all files

63
beacon_node/network/src/sync/range_sync/chain.rs
View File

@@ -11,7 +11,6 @@ use slog::{crit, debug, o, warn};
use std::collections::{btree_map::Entry, BTreeMap, HashSet};
use std::hash::{Hash, Hasher};
use std::sync::Arc;
use tokio::sync::mpsc::Sender;
use types::{Epoch, EthSpec, Hash256, SignedBeaconBlock, Slot};
/// Blocks are downloaded in batches from peers. This constant specifies how many epochs worth of
@@ -102,9 +101,6 @@ pub struct SyncingChain<T: BeaconChainTypes> {
/// Batches validated by this chain.
validated_batches: u64,
/// A multi-threaded, non-blocking processor for applying messages to the beacon chain.
beacon_processor_send: Sender<BeaconWorkEvent<T>>,
is_finalized_segment: bool,
/// The chain's log.
@@ -132,7 +128,6 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
target_head_slot: Slot,
target_head_root: Hash256,
peer_id: PeerId,
beacon_processor_send: Sender<BeaconWorkEvent<T>>,
is_finalized_segment: bool,
log: &slog::Logger,
) -> Self {
@@ -155,7 +150,6 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
state: ChainSyncingState::Stopped,
current_processing_batch: None,
validated_batches: 0,
beacon_processor_send,
is_finalized_segment,
log: log.new(o!("chain" => id)),
}
@@ -186,7 +180,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
pub fn remove_peer(
&mut self,
peer_id: &PeerId,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
) -> ProcessingResult {
if let Some(batch_ids) = self.peers.remove(peer_id) {
// fail the batches
@@ -227,7 +221,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// If the block correctly completes the batch it will be processed if possible.
pub fn on_block_response(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
peer_id: &PeerId,
request_id: Id,
@@ -296,7 +290,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// The batch must exist and be ready for processing
fn process_batch(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
) -> ProcessingResult {
// Only process batches if this chain is Syncing, and only one at a time
@@ -304,6 +298,11 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
return Ok(KeepChain);
}
let beacon_processor_send = match network.processor_channel_if_enabled() {
Some(channel) => channel,
None => return Ok(KeepChain),
};
let batch = match self.batches.get_mut(&batch_id) {
Some(batch) => batch,
None => {
@@ -327,9 +326,8 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
let process_id = ChainSegmentProcessId::RangeBatchId(self.id, batch_id, count_unrealized);
self.current_processing_batch = Some(batch_id);
if let Err(e) = self
.beacon_processor_send
.try_send(BeaconWorkEvent::chain_segment(process_id, blocks))
if let Err(e) =
beacon_processor_send.try_send(BeaconWorkEvent::chain_segment(process_id, blocks))
{
crit!(self.log, "Failed to send chain segment to processor."; "msg" => "process_batch",
"error" => %e, "batch" => self.processing_target);
@@ -346,7 +344,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// Processes the next ready batch, prioritizing optimistic batches over the processing target.
fn process_completed_batches(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
) -> ProcessingResult {
// Only process batches if this chain is Syncing and only process one batch at a time
if self.state != ChainSyncingState::Syncing || self.current_processing_batch.is_some() {
@@ -447,7 +445,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// of the batch processor.
pub fn on_batch_process_result(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
result: &BatchProcessResult,
) -> ProcessingResult {
@@ -580,7 +578,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
fn reject_optimistic_batch(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
redownload: bool,
reason: &str,
) -> ProcessingResult {
@@ -611,11 +609,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
///
/// If a previous batch has been validated and it had been re-processed, penalize the original
/// peer.
fn advance_chain(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
validating_epoch: Epoch,
) {
fn advance_chain(&mut self, network: &mut SyncNetworkContext<T>, validating_epoch: Epoch) {
// make sure this epoch produces an advancement
if validating_epoch <= self.start_epoch {
return;
@@ -719,7 +713,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// intended and can result in downvoting a peer.
fn handle_invalid_batch(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
) -> ProcessingResult {
// The current batch could not be processed, indicating either the current or previous
@@ -778,7 +772,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// This could be new chain, or an old chain that is being resumed.
pub fn start_syncing(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
local_finalized_epoch: Epoch,
optimistic_start_epoch: Epoch,
) -> ProcessingResult {
@@ -816,7 +810,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// If the chain is active, this starts requesting batches from this peer.
pub fn add_peer(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
peer_id: PeerId,
) -> ProcessingResult {
// add the peer without overwriting its active requests
@@ -833,7 +827,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// If the batch exists it is re-requested.
pub fn inject_error(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
peer_id: &PeerId,
request_id: Id,
@@ -865,7 +859,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// Sends and registers the request of a batch awaiting download.
pub fn retry_batch_download(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
) -> ProcessingResult {
let batch = match self.batches.get_mut(&batch_id) {
@@ -898,7 +892,7 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
/// Requests the batch assigned to the given id from a given peer.
pub fn send_batch(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
network: &mut SyncNetworkContext<T>,
batch_id: BatchId,
peer: PeerId,
) -> ProcessingResult {
@@ -967,12 +961,21 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
}
}
/// Kickstarts the chain by sending for processing batches that are ready and requesting more
/// batches if needed.
pub fn resume(
&mut self,
network: &mut SyncNetworkContext<T>,
) -> Result<KeepChain, RemoveChain> {
// Request more batches if needed.
self.request_batches(network)?;
// If there is any batch ready for processing, send it.
self.process_completed_batches(network)
}
/// Attempts to request the next required batches from the peer pool if the chain is syncing. It will exhaust the peer
/// pool and left over batches until the batch buffer is reached or all peers are exhausted.
fn request_batches(
&mut self,
network: &mut SyncNetworkContext<T::EthSpec>,
) -> ProcessingResult {
fn request_batches(&mut self, network: &mut SyncNetworkContext<T>) -> ProcessingResult {
if !matches!(self.state, ChainSyncingState::Syncing) {
return Ok(KeepChain);
}