Write validator definitions atomically (#2338)

## Issue Addressed

Closes https://github.com/sigp/lighthouse/issues/2159

## Proposed Changes

Rather than trying to write the validator definitions to disk directly, use a temporary file called `.validator_defintions.yml.tmp` and then atomically rename it to `validator_definitions.yml`. This avoids truncating the primary file, which can cause permanent damage when the disk is full.

The same treatment is also applied to the validator key cache, although the situation is less dire if it becomes corrupted because it can just be deleted without the user having to reimport keys or resupply passwords.

## Additional Info

* `File::create` truncates upon opening: https://doc.rust-lang.org/std/fs/struct.File.html#method.create
* `fs::rename` uses `rename` on UNIX and `MoveFileEx` on Windows: https://doc.rust-lang.org/std/fs/fn.rename.html
* UNIX `rename` call is atomic: https://unix.stackexchange.com/questions/322038/is-mv-atomic-on-my-fs
* Windows `MoveFileEx` is _not_ atomic in general, and Windows lacks any clear API for atomic file renames :(
   https://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows

## Further Work

* Consider whether we want to try a different Windows syscall as part of #2333. The `rust-atomicwrites` crate seems promising, but actually uses the same syscall under the hood presently: https://github.com/untitaker/rust-atomicwrites/issues/27.

validator_client/src/key_cache.rs

@@ -1,4 +1,4 @@
-use account_utils::create_with_600_perms;
+use account_utils::write_file_via_temporary;
 use bls::{Keypair, PublicKey};
 use eth2_keystore::json_keystore::{
     Aes128Ctr, ChecksumModule, Cipher, CipherModule, Crypto, EmptyMap, EmptyString, KdfModule,
@@ -12,12 +12,15 @@ use rand::prelude::*;
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
 use std::fs::OpenOptions;
+use std::io;
 use std::path::{Path, PathBuf};
-use std::{fs, io};
 
 /// The file name for the serialized `KeyCache` struct.
 pub const CACHE_FILENAME: &str = "validator_key_cache.json";
 
+/// The file name for the temporary `KeyCache`.
+pub const TEMP_CACHE_FILENAME: &str = ".validator_key_cache.json.tmp";
+
 #[derive(Debug, Copy, Clone, PartialEq)]
 pub enum State {
     NotDecrypted,
@@ -139,17 +142,14 @@ impl KeyCache {
             self.encrypt()?;
 
             let cache_path = validators_dir.as_ref().join(CACHE_FILENAME);
+            let temp_path = validators_dir.as_ref().join(TEMP_CACHE_FILENAME);
             let bytes = serde_json::to_vec(self).map_err(Error::UnableToEncodeFile)?;
 
-            let res = if cache_path.exists() {
-                fs::write(cache_path, &bytes).map_err(Error::UnableToWriteFile)
-            } else {
-                create_with_600_perms(&cache_path, &bytes).map_err(Error::UnableToWriteFile)
-            };
-            if res.is_ok() {
-                self.state = State::DecryptedAndSaved;
-            }
-            res.map(|_| true)
+            write_file_via_temporary(&cache_path, &temp_path, &bytes)
+                .map_err(Error::UnableToWriteFile)?;
+
+            self.state = State::DecryptedAndSaved;
+            Ok(true)
         } else {
             Ok(false)
         }
@@ -243,7 +243,7 @@ pub enum Error {
     UnableToParseFile(serde_json::Error),
     /// The cache file could not be serialized as YAML.
     UnableToEncodeFile(serde_json::Error),
-    /// The cache file could not be written to the filesystem.
+    /// The cache file or its temporary could not be written to the filesystem.
     UnableToWriteFile(io::Error),
     /// Couldn't decrypt the cache file
     UnableToDecrypt(KeystoreError),