gnosis/lighthouse
1
0
Fork 0
mirror of https://github.com/sigp/lighthouse.git synced 2026-05-08 01:05:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

Implement SSZ union type (#2579)

Browse Source 
## Issue Addressed

NA

## Proposed Changes

Implements the "union" type from the SSZ spec for `ssz`, `ssz_derive`, `tree_hash` and `tree_hash_derive` so it may be derived for `enums`:

https://github.com/ethereum/consensus-specs/blob/v1.1.0-beta.3/ssz/simple-serialize.md#union

The union type is required for the merge, since the `Transaction` type is defined as a single-variant union `Union[OpaqueTransaction]`.

### Crate Updates

This PR will (hopefully) cause CI to publish new versions for the following crates:

- `eth2_ssz_derive`: `0.2.1` -> `0.3.0`
- `eth2_ssz`: `0.3.0` -> `0.4.0`
- `eth2_ssz_types`: `0.2.0` -> `0.2.1`
- `tree_hash`: `0.3.0` -> `0.4.0`
- `tree_hash_derive`: `0.3.0` -> `0.4.0`

These these crates depend on each other, I've had to add a workspace-level `[patch]` for these crates. A follow-up PR will need to remove this patch, ones the new versions are published.

### Union Behaviors

We already had SSZ `Encode` and `TreeHash` derive for enums, however it just did a "transparent" pass-through of the inner value. Since the "union" decoding from the spec is in conflict with the transparent method, I've required that all `enum` have exactly one of the following enum-level attributes:

#### SSZ

-  `#[ssz(enum_behaviour = "union")]`
    - matches the spec used for the merge
-  `#[ssz(enum_behaviour = "transparent")]`
    - maintains existing functionality
    - not supported for `Decode` (never was)
    
#### TreeHash

-  `#[tree_hash(enum_behaviour = "union")]`
    - matches the spec used for the merge
-  `#[tree_hash(enum_behaviour = "transparent")]`
    - maintains existing functionality

This means that we can maintain the existing transparent behaviour, but all existing users will get a compile-time error until they explicitly opt-in to being transparent.

### Legacy Option Encoding

Before this PR, we already had a union-esque encoding for `Option<T>`. However, this was with the *old* SSZ spec where the union selector was 4 bytes. During merge specification, the spec was changed to use 1 byte for the selector.

Whilst the 4-byte `Option` encoding was never used in the spec, we used it in our database. Writing a migrate script for all occurrences of `Option` in the database would be painful, especially since it's used in the `CommitteeCache`. To avoid the migrate script, I added a serde-esque `#[ssz(with = "module")]` field-level attribute to `ssz_derive` so that we can opt into the 4-byte encoding on a field-by-field basis.

The `ssz::legacy::four_byte_impl!` macro allows a one-liner to define the module required for the `#[ssz(with = "module")]` for some `Option<T> where T: Encode + Decode`.

Notably, **I have removed `Encode` and `Decode` impls for `Option`**. I've done this to force a break on downstream users. Like I mentioned, `Option` isn't used in the spec so I don't think it'll be *that* annoying. I think it's nicer than quietly having two different union implementations or quietly breaking the existing `Option` impl.

### Crate Publish Ordering

I've modified the order in which CI publishes crates to ensure that we don't publish a crate without ensuring we already published a crate that it depends upon.

## TODO

- [ ] Queue a follow-up `[patch]`-removing PR.
This commit is contained in:
Paul Hauner
2021-09-25 05:58:36 +00:00
parent a844ce5ba9
commit fe52322088
63 changed files with 1515 additions and 571 deletions
Download Patch File Download Diff File Expand all files Collapse all files

265
consensus/ssz/src/legacy.rs Normal file
View File

@@ -0,0 +1,265 @@
//! Provides a "legacy" version of SSZ encoding for `Option<T> where T: Encode + Decode`.
//!
//! The SSZ specification changed in 2021 to use a 1-byte union selector, instead of a 4-byte one
//! which was used in the Lighthouse database.
//!
//! Users can use the `four_byte_option_impl` macro to define a module that can be used with the
//! `#[ssz(with = "module")]`.
//!
//! ## Example
//!
//! ```rust
//! use ssz_derive::{Encode, Decode};
//! use ssz::four_byte_option_impl;
//!
//! four_byte_option_impl!(impl_for_u64, u64);
//!
//! #[derive(Encode, Decode)]
//! struct Foo {
//! #[ssz(with = "impl_for_u64")]
//! a: Option<u64>,
//! }
//! ```
use crate::*;
#[macro_export]
macro_rules! four_byte_option_impl {
($mod_name: ident, $type: ty) => {
#[allow(dead_code)]
mod $mod_name {
use super::*;
pub mod encode {
use super::*;
#[allow(unused_imports)]
use ssz::*;
pub fn is_ssz_fixed_len() -> bool {
false
}
pub fn ssz_fixed_len() -> usize {
BYTES_PER_LENGTH_OFFSET
}
pub fn ssz_bytes_len(opt: &Option<$type>) -> usize {
if let Some(some) = opt {
let len = if <$type as Encode>::is_ssz_fixed_len() {
<$type as Encode>::ssz_fixed_len()
} else {
<$type as Encode>::ssz_bytes_len(some)
};
len + BYTES_PER_LENGTH_OFFSET
} else {
BYTES_PER_LENGTH_OFFSET
}
}
pub fn ssz_append(opt: &Option<$type>, buf: &mut Vec<u8>) {
match opt {
None => buf.extend_from_slice(&legacy::encode_four_byte_union_selector(0)),
Some(t) => {
buf.extend_from_slice(&legacy::encode_four_byte_union_selector(1));
t.ssz_append(buf);
}
}
}
pub fn as_ssz_bytes(opt: &Option<$type>) -> Vec<u8> {
let mut buf = vec![];
ssz_append(opt, &mut buf);
buf
}
}
pub mod decode {
use super::*;
#[allow(unused_imports)]
use ssz::*;
pub fn is_ssz_fixed_len() -> bool {
false
}
pub fn ssz_fixed_len() -> usize {
BYTES_PER_LENGTH_OFFSET
}
pub fn from_ssz_bytes(bytes: &[u8]) -> Result<Option<$type>, DecodeError> {
if bytes.len() < BYTES_PER_LENGTH_OFFSET {
return Err(DecodeError::InvalidByteLength {
len: bytes.len(),
expected: BYTES_PER_LENGTH_OFFSET,
});
}
let (index_bytes, value_bytes) = bytes.split_at(BYTES_PER_LENGTH_OFFSET);
let index = legacy::read_four_byte_union_selector(index_bytes)?;
if index == 0 {
Ok(None)
} else if index == 1 {
Ok(Some(<$type as ssz::Decode>::from_ssz_bytes(value_bytes)?))
} else {
Err(DecodeError::BytesInvalid(format!(
"{} is not a valid union index for Option<T>",
index
)))
}
}
}
}
};
}
pub fn encode_four_byte_union_selector(selector: usize) -> [u8; BYTES_PER_LENGTH_OFFSET] {
encode_length(selector)
}
pub fn read_four_byte_union_selector(bytes: &[u8]) -> Result<usize, DecodeError> {
read_offset(bytes)
}
#[cfg(test)]
mod test {
use super::*;
use crate as ssz;
use ssz_derive::{Decode, Encode};
type VecU16 = Vec<u16>;
four_byte_option_impl!(impl_u16, u16);
four_byte_option_impl!(impl_vec_u16, VecU16);
#[test]
fn ssz_encode_option_u16() {
let item = Some(65535_u16);
let bytes = vec![1, 0, 0, 0, 255, 255];
assert_eq!(impl_u16::encode::as_ssz_bytes(&item), bytes);
assert_eq!(impl_u16::decode::from_ssz_bytes(&bytes).unwrap(), item);
let item = None;
let bytes = vec![0, 0, 0, 0];
assert_eq!(impl_u16::encode::as_ssz_bytes(&item), bytes);
assert_eq!(impl_u16::decode::from_ssz_bytes(&bytes).unwrap(), None);
}
#[test]
fn ssz_encode_option_vec_u16() {
let item = Some(vec![0_u16, 1]);
let bytes = vec![1, 0, 0, 0, 0, 0, 1, 0];
assert_eq!(impl_vec_u16::encode::as_ssz_bytes(&item), bytes);
assert_eq!(impl_vec_u16::decode::from_ssz_bytes(&bytes).unwrap(), item);
let item = None;
let bytes = vec![0, 0, 0, 0];
assert_eq!(impl_vec_u16::encode::as_ssz_bytes(&item), bytes);
assert_eq!(impl_vec_u16::decode::from_ssz_bytes(&bytes).unwrap(), item);
}
fn round_trip<T: Encode + Decode + std::fmt::Debug + PartialEq>(items: Vec<T>) {
for item in items {
let encoded = &item.as_ssz_bytes();
assert_eq!(item.ssz_bytes_len(), encoded.len());
assert_eq!(T::from_ssz_bytes(encoded), Ok(item));
}
}
#[derive(Debug, PartialEq, Encode, Decode)]
struct TwoVariableLenOptions {
a: u16,
#[ssz(with = "impl_u16")]
b: Option<u16>,
#[ssz(with = "impl_vec_u16")]
c: Option<Vec<u16>>,
#[ssz(with = "impl_vec_u16")]
d: Option<Vec<u16>>,
}
#[test]
#[allow(clippy::zero_prefixed_literal)]
fn two_variable_len_options_encoding() {
let s = TwoVariableLenOptions {
a: 42,
b: None,
c: Some(vec![0]),
d: None,
};
let bytes = vec![
// 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
// | option<u16> | offset | offset | option<u16 | 1st list
42, 00, 14, 00, 00, 00, 18, 00, 00, 00, 24, 00, 00, 00, 00, 00, 00, 00, 01, 00, 00, 00,
// 23 24 25 26 27
// | 2nd list
00, 00, 00, 00, 00, 00,
];
assert_eq!(s.as_ssz_bytes(), bytes);
}
#[test]
fn two_variable_len_options_round_trip() {
let vec: Vec<TwoVariableLenOptions> = vec![
TwoVariableLenOptions {
a: 42,
b: Some(12),
c: Some(vec![0]),
d: Some(vec![1]),
},
TwoVariableLenOptions {
a: 42,
b: Some(12),
c: Some(vec![0]),
d: None,
},
TwoVariableLenOptions {
a: 42,
b: None,
c: Some(vec![0]),
d: None,
},
TwoVariableLenOptions {
a: 42,
b: None,
c: None,
d: None,
},
];
round_trip(vec);
}
#[test]
fn tuple_u8_u16() {
let vec: Vec<(u8, u16)> = vec![
(0, 0),
(0, 1),
(1, 0),
(u8::max_value(), u16::max_value()),
(0, u16::max_value()),
(u8::max_value(), 0),
(42, 12301),
];
round_trip(vec);
}
#[test]
fn tuple_vec_vec() {
let vec: Vec<(u64, Vec<u8>, Vec<Vec<u16>>)> = vec![
(0, vec![], vec![vec![]]),
(99, vec![101], vec![vec![], vec![]]),
(
42,
vec![12, 13, 14],
vec![vec![99, 98, 97, 96], vec![42, 44, 46, 48, 50]],
),
];
round_trip(vec);
}
}