Separate BN for block proposals (#4182)

It is a well-known fact that IP addresses for beacon nodes used by specific validators can be de-anonymized. There is an assumed risk that a malicious user may attempt to DOS validators when producing blocks to prevent chain growth/liveness.

Although there are a number of ideas put forward to address this, there a few simple approaches we can take to mitigate this risk.

Currently, a Lighthouse user is able to set a number of beacon-nodes that their validator client can connect to. If one beacon node is taken offline, it can fallback to another. Different beacon nodes can use VPNs or rotate IPs in order to mask their IPs.

This PR provides an additional setup option which further mitigates attacks of this kind.

This PR introduces a CLI flag --proposer-only to the beacon node. Setting this flag will configure the beacon node to run with minimal peers and crucially will not subscribe to subnets or sync committees. Therefore nodes of this kind should not be identified as nodes connected to validators of any kind.

It also introduces a CLI flag --proposer-nodes to the validator client. Users can then provide a number of beacon nodes (which may or may not run the --proposer-only flag) that the Validator client will use for block production and propagation only. If these nodes fail, the validator client will fallback to the default list of beacon nodes.

Users are then able to set up a number of beacon nodes dedicated to block proposals (which are unlikely to be identified as validator nodes) and point their validator clients to produce blocks on these nodes and attest on other beacon nodes. An attack attempting to prevent liveness on the eth2 network would then need to preemptively find and attack the proposer nodes which is significantly more difficult than the default setup.

This is a follow on from: #3328 

Co-authored-by: Michael Sproul <michael@sigmaprime.io>
Co-authored-by: Paul Hauner <paul@paulhauner.com>

testing/simulator/src/cli.rs

@@ -24,6 +24,12 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                         .takes_value(true)
                         .default_value("4")
                         .help("Number of beacon nodes"))
+                    .arg(Arg::with_name("proposer-nodes")
+                        .short("n")
+                        .long("nodes")
+                        .takes_value(true)
+                        .default_value("2")
+                        .help("Number of proposer-only beacon nodes"))
                     .arg(Arg::with_name("validators_per_node")
                         .short("v")
                         .long("validators_per_node")
@@ -57,6 +63,12 @@ pub fn cli_app<'a, 'b>() -> App<'a, 'b> {
                         .takes_value(true)
                         .default_value("4")
                         .help("Number of beacon nodes"))
+                    .arg(Arg::with_name("proposer-nodes")
+                        .short("n")
+                        .long("nodes")
+                        .takes_value(true)
+                        .default_value("2")
+                        .help("Number of proposer-only beacon nodes"))
                     .arg(Arg::with_name("validators_per_node")
                         .short("v")
                         .long("validators_per_node")

testing/simulator/src/eth1_sim.rs

@@ -27,6 +27,8 @@ const SUGGESTED_FEE_RECIPIENT: [u8; 20] =
 
 pub fn run_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
     let node_count = value_t!(matches, "nodes", usize).expect("missing nodes default");
+    let proposer_nodes = value_t!(matches, "proposer-nodes", usize).unwrap_or(0);
+    println!("PROPOSER-NODES: {}", proposer_nodes);
     let validators_per_node = value_t!(matches, "validators_per_node", usize)
         .expect("missing validators_per_node default");
     let speed_up_factor =
@@ -35,7 +37,8 @@ pub fn run_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
     let post_merge_sim = matches.is_present("post-merge");
 
     println!("Beacon Chain Simulator:");
-    println!(" nodes:{}", node_count);
+    println!(" nodes:{}, proposer_nodes: {}", node_count, proposer_nodes);
+
     println!(" validators_per_node:{}", validators_per_node);
     println!(" post merge simulation:{}", post_merge_sim);
     println!(" continue_after_checks:{}", continue_after_checks);
@@ -147,7 +150,7 @@ pub fn run_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
         beacon_config.sync_eth1_chain = true;
         beacon_config.eth1.auto_update_interval_millis = eth1_block_time.as_millis() as u64;
         beacon_config.eth1.chain_id = Eth1Id::from(chain_id);
-        beacon_config.network.target_peers = node_count - 1;
+        beacon_config.network.target_peers = node_count + proposer_nodes - 1;
 
         beacon_config.network.enr_address = (Some(Ipv4Addr::LOCALHOST), None);
 
@@ -173,7 +176,17 @@ pub fn run_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
          * One by one, add beacon nodes to the network.
          */
         for _ in 0..node_count - 1 {
-            network.add_beacon_node(beacon_config.clone()).await?;
+            network
+                .add_beacon_node(beacon_config.clone(), false)
+                .await?;
+        }
+
+        /*
+         * One by one, add proposer nodes to the network.
+         */
+        for _ in 0..proposer_nodes - 1 {
+            println!("Adding a proposer node");
+            network.add_beacon_node(beacon_config.clone(), true).await?;
         }
 
         /*
@@ -310,7 +323,7 @@ pub fn run_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
          */
         println!(
             "Simulation complete. Finished with {} beacon nodes and {} validator clients",
-            network.beacon_node_count(),
+            network.beacon_node_count() + network.proposer_node_count(),
             network.validator_client_count()
         );
 

testing/simulator/src/local_network.rs

@@ -25,6 +25,7 @@ pub const TERMINAL_BLOCK: u64 = 64;
 pub struct Inner<E: EthSpec> {
     pub context: RuntimeContext<E>,
     pub beacon_nodes: RwLock<Vec<LocalBeaconNode<E>>>,
+    pub proposer_nodes: RwLock<Vec<LocalBeaconNode<E>>>,
     pub validator_clients: RwLock<Vec<LocalValidatorClient<E>>>,
     pub execution_nodes: RwLock<Vec<LocalExecutionNode<E>>>,
 }
@@ -97,6 +98,7 @@ impl<E: EthSpec> LocalNetwork<E> {
             inner: Arc::new(Inner {
                 context,
                 beacon_nodes: RwLock::new(vec![beacon_node]),
+                proposer_nodes: RwLock::new(vec![]),
                 execution_nodes: RwLock::new(execution_node),
                 validator_clients: RwLock::new(vec![]),
             }),
@@ -111,6 +113,14 @@ impl<E: EthSpec> LocalNetwork<E> {
         self.beacon_nodes.read().len()
     }
 
+    /// Returns the number of proposer nodes in the network.
+    ///
+    /// Note: does not count nodes that are external to this `LocalNetwork` that may have connected
+    /// (e.g., another Lighthouse process on the same machine.)
+    pub fn proposer_node_count(&self) -> usize {
+        self.proposer_nodes.read().len()
+    }
+
     /// Returns the number of validator clients in the network.
     ///
     /// Note: does not count nodes that are external to this `LocalNetwork` that may have connected
@@ -120,7 +130,11 @@ impl<E: EthSpec> LocalNetwork<E> {
     }
 
     /// Adds a beacon node to the network, connecting to the 0'th beacon node via ENR.
-    pub async fn add_beacon_node(&self, mut beacon_config: ClientConfig) -> Result<(), String> {
+    pub async fn add_beacon_node(
+        &self,
+        mut beacon_config: ClientConfig,
+        is_proposer: bool,
+    ) -> Result<(), String> {
         let self_1 = self.clone();
         let count = self.beacon_node_count() as u16;
         println!("Adding beacon node..");
@@ -135,6 +149,7 @@ impl<E: EthSpec> LocalNetwork<E> {
                     .enr()
                     .expect("bootnode must have a network"),
             );
+            let count = (self.beacon_node_count() + self.proposer_node_count()) as u16;
             beacon_config.network.set_ipv4_listening_address(
                 std::net::Ipv4Addr::UNSPECIFIED,
                 BOOTNODE_PORT + count,
@@ -143,6 +158,7 @@ impl<E: EthSpec> LocalNetwork<E> {
             beacon_config.network.enr_udp4_port = Some(BOOTNODE_PORT + count);
             beacon_config.network.enr_tcp4_port = Some(BOOTNODE_PORT + count);
             beacon_config.network.discv5_config.table_filter = |_| true;
+            beacon_config.network.proposer_only = is_proposer;
         }
         if let Some(el_config) = &mut beacon_config.execution_layer {
             let config = MockExecutionConfig {
@@ -173,7 +189,11 @@ impl<E: EthSpec> LocalNetwork<E> {
             beacon_config,
         )
         .await?;
-        self_1.beacon_nodes.write().push(beacon_node);
+        if is_proposer {
+            self_1.proposer_nodes.write().push(beacon_node);
+        } else {
+            self_1.beacon_nodes.write().push(beacon_node);
+        }
         Ok(())
     }
 
@@ -200,6 +220,16 @@ impl<E: EthSpec> LocalNetwork<E> {
                 .http_api_listen_addr()
                 .expect("Must have http started")
         };
+        // If there is a proposer node for the same index, we will use that for proposing
+        let proposer_socket_addr = {
+            let read_lock = self.proposer_nodes.read();
+            read_lock.get(beacon_node).map(|proposer_node| {
+                proposer_node
+                    .client
+                    .http_api_listen_addr()
+                    .expect("Must have http started")
+            })
+        };
 
         let beacon_node = SensitiveUrl::parse(
             format!("http://{}:{}", socket_addr.ip(), socket_addr.port()).as_str(),
@@ -210,6 +240,21 @@ impl<E: EthSpec> LocalNetwork<E> {
         } else {
             vec![beacon_node]
         };
+
+        // If we have a proposer node established, use it.
+        if let Some(proposer_socket_addr) = proposer_socket_addr {
+            let url = SensitiveUrl::parse(
+                format!(
+                    "http://{}:{}",
+                    proposer_socket_addr.ip(),
+                    proposer_socket_addr.port()
+                )
+                .as_str(),
+            )
+            .unwrap();
+            validator_config.proposer_nodes = vec![url];
+        }
+
         let validator_client = LocalValidatorClient::production_with_insecure_keypairs(
             context,
             validator_config,
@@ -223,9 +268,11 @@ impl<E: EthSpec> LocalNetwork<E> {
     /// For all beacon nodes in `Self`, return a HTTP client to access each nodes HTTP API.
     pub fn remote_nodes(&self) -> Result<Vec<BeaconNodeHttpClient>, String> {
         let beacon_nodes = self.beacon_nodes.read();
+        let proposer_nodes = self.proposer_nodes.read();
 
         beacon_nodes
             .iter()
+            .chain(proposer_nodes.iter())
             .map(|beacon_node| beacon_node.remote_node())
             .collect()
     }

testing/simulator/src/no_eth1_sim.rs

@@ -100,7 +100,9 @@ pub fn run_no_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
          */
 
         for _ in 0..node_count - 1 {
-            network.add_beacon_node(beacon_config.clone()).await?;
+            network
+                .add_beacon_node(beacon_config.clone(), false)
+                .await?;
         }
 
         /*
@@ -151,7 +153,7 @@ pub fn run_no_eth1_sim(matches: &ArgMatches) -> Result<(), String> {
          */
         println!(
             "Simulation complete. Finished with {} beacon nodes and {} validator clients",
-            network.beacon_node_count(),
+            network.beacon_node_count() + network.proposer_node_count(),
             network.validator_client_count()
         );
 

testing/simulator/src/sync_sim.rs

@@ -228,7 +228,7 @@ pub async fn verify_one_node_sync<E: EthSpec>(
     )
     .await;
     // Add a beacon node
-    network.add_beacon_node(beacon_config).await?;
+    network.add_beacon_node(beacon_config, false).await?;
     // Check every `epoch_duration` if nodes are synced
     // limited to at most `sync_timeout` epochs
     let mut interval = tokio::time::interval(epoch_duration);
@@ -265,8 +265,10 @@ pub async fn verify_two_nodes_sync<E: EthSpec>(
     )
     .await;
     // Add beacon nodes
-    network.add_beacon_node(beacon_config.clone()).await?;
-    network.add_beacon_node(beacon_config).await?;
+    network
+        .add_beacon_node(beacon_config.clone(), false)
+        .await?;
+    network.add_beacon_node(beacon_config, false).await?;
     // Check every `epoch_duration` if nodes are synced
     // limited to at most `sync_timeout` epochs
     let mut interval = tokio::time::interval(epoch_duration);
@@ -305,8 +307,10 @@ pub async fn verify_in_between_sync<E: EthSpec>(
     )
     .await;
     // Add two beacon nodes
-    network.add_beacon_node(beacon_config.clone()).await?;
-    network.add_beacon_node(beacon_config).await?;
+    network
+        .add_beacon_node(beacon_config.clone(), false)
+        .await?;
+    network.add_beacon_node(beacon_config, false).await?;
     // Delay before adding additional syncing nodes.
     epoch_delay(
         Epoch::new(sync_timeout - 5),
@@ -315,7 +319,7 @@ pub async fn verify_in_between_sync<E: EthSpec>(
     )
     .await;
     // Add a beacon node
-    network.add_beacon_node(config1.clone()).await?;
+    network.add_beacon_node(config1.clone(), false).await?;
     // Check every `epoch_duration` if nodes are synced
     // limited to at most `sync_timeout` epochs
     let mut interval = tokio::time::interval(epoch_duration);