Technical Release notes and Changelog for 0.7.6

#889 NO-TICKET: Use timestamp in candidate block hash.

https://github.com/CasperLabs/casper-node/pull/889

#887 NO TICKET: Do not expose private method

https://github.com/CasperLabs/casper-node/pull/887

#883 NO-TICKET: Reduce consensus broadcast traffic

The consensus component currently uses broadcast as a temporary replacement for gossip, and re-broadcasts every received vertex, because it was assumed that that's how gossip would work.

With small_network, there is no gossip, and thus this creates a lot of redundant network traffic.
With libp2p, gossiping will work differently, and explicitly distinguish between sending a new vertex and evaluating an incoming one before re-gossiping it.

So with neither of the two networking implementations it makes sense to return a NewVertex effect for an incoming vertex. This PR removes that effect.

#882 HWY-251: BlockValidator accepts alternative sources of data.

After this PR, if fetching a Deploy fails due to a timeout then BlockValidator will retry downloading a deploy from an alternative source.

#881 HWY-250: Use `ConsensusValue`'s hash as a key in map.

Instead of storing the whole ConsensusValue as a key in the map (where ConsensusValue is a set of all deploys that the block includes), we will store a hash of ConsensusValue. This will improve memory management.

#880 HWY-246: Deduplicate synchronizer entries.

We are seeing high memory usage in tests, due to pending vertices in the Highway synchronizer. Currently a new entry in a synchronizer queue is created whenever we receive a vertex that we can't add to the protocol state yet, e.g. due to missing dependencies. If we receive the same vertex multiple times, it will be duplicated in multiple queue entries.

This PR turns the queues into maps, keyed by the vertex and the sending peer, and when we receive a duplicate it only updates the timestamp of the existing entry instead of creating a duplicate.

#879 HWY-249: Add the timestamp to `CandidateBlock`

Validity of a proto block depends on its timestamp. However, when a particular proto block gets validated, the consensus component doesn't remember its timestamp, and if the same proto block was proposed at two different places, it applies the validation result to both.

Let's put the timestamp into CandidateBlock.

#877 NO-TICKET: Fixes a bug in `BlockLike` implementation for `Block` and adds logging to the `BlockValidator`

#876 NO-TICKET/networking metrics

This adds generic networking metrics, counting queued message and requests for outgoing broadcasts, as well as direct messages.

Currently only implemented for small_net due to some time constraints, but the same structure should be used for the libp2p networking component. Having the same metrics on both will allow for some comparability.

These metrics should be especially useful when tweaking the chattiness of consensus.

#875 DRS-739: Testing the joiner reactor

#874 NDRS-860: include next upgrade activation point in status endpoints

This PR provides the next upgrade activation point in the status endpoints.

Manual testing via nctl showed that the status endpoints showed correct values for the activation point (null at start, then a valid era ID, then an updated era ID).

#873 NDRS-840: replace hard-coded prococol versions

This PR provides the next upgrade activation point in the status endpoints.

Manual testing via nctl showed that the status endpoints showed correct values for the activation point (null at start, then a valid era ID, then an updated era ID).

#872 NO-TICKET: fix short-circuit in Runtime::transfer_to_account

No functional changes here, just noticed that this check wasn't correctly factoring in access rights so that we weren't actually short-circuiting when transferring from a given source to itself.

#871 ECO-863: add faucet tests

This PR validates that recent changes to the faucet contract preserved the existing behavior.

#870 NO-TICK: Reuse LMDB txn

LMDB is configured with MAX_TRANSACTIONS = 1 but we tried to create second txn without dropping the first one.

#869 NO-TICKET: Revert changes in finality signatures

Reverted changes made by NDRS-767: Remove Finality Signatures from Block

#868 NO-TICKET: More detailed preconditions.

This PR changes StepResult::PreconditionError and splits it into separate, more detailed variants that also holds the error instance.

#867 NO-TICK: Changes for fixing crates publish on tag

Removed update-revision to stop changes to cargo.toml.
Added package-tag flag to be used in tagging pipeline.

#866 NCTL :: Test account pre-funding

Ensures user accounts are pre-funded.

#865 NO-TICK: Store new finality signatures in the storage.

As part of the work on NDRS-833 I've noticed that we don't store new finality signatures in the persistent storage. This PR fixes that problem.

#863 Output detailed consensus component memory metrics in log

#862 NDRS-841: avoid generating the client ffi header in the source tree

This PR makes bindgen generate the C header of our client FFI in the project's target folder rather than the client's source tree. This allows cargo publish to function as expected.

#861 EE-1164: add vesting schedule for validator bids

Implements the following logic for genesis stakers. “The Purchaser agrees that its Purchased Tokens shall remain Staked with respect to the CasperLabs Blockchain, and thus not transferable, for the ninety (90) days following the MainNet Launch, and the Purchaser acknowledges that its Purchased Tokens shall be partially released and made transferable (i.e., a portion of the Purchased Tokens shall be unbonded) only in accordance with the protocol validator rotation and unbonding wait period, as set forth in the Software, and which is expected to allow the Purchaser to unbond its Purchased Tokens from the CasperLabs Blockchain in equal amounts each week over the period of 90 days, commencing ninety (90) days following the MainNet Launch (the “Purchased Tokens Release Schedule”). The Company shall ensure that all Tokens purchased in the Validator Sale are subject to the release schedule same as the Purchased Tokens Release Schedule.”

#859 Changing settings for throttling before 0.7.4

#858 OP-1807: adding CARGO_FLAGS to cargo-test

Sets CARGO_FLAGS=--release for the cargo-test step.

#857 NDRS-842: Have network retry connecting

#856 HWY-242: Upgrade DataSize; remove skips.

#855 HWY-242: Fix DataSize implementation for HighwayProtocol

The DataSize implementation for HighwayProtocol counts all HighwayProtocol instances as 0, because it uses the DataSize implementation for a reference.estimate_heap_size was called on &HighwayProtocol instead of HighwayProtocol and returned 0.

#851 NO-TICKET: replace yamux with mplex

This replaces the yamux multiplexer in the libp2p network component with an mplex one. This change was observed to significantly lower the memory usage of test network. The PR also temporarily disables the send_large_message_across_network test while investigations proceed into its sporadic failure.

#850 Cleaning up and simplifying

Readme and documentation updates.

#849 NO-TICKET: update RPC server

This PR brings the JSON-RPC server more in line with the error object as defined in the specs.

• -32601, "Method not found" is now handled correctly

• -32602, "Invalid params" is now handled correctly

• -32603, "Internal error" is not required

• our server error codes have been negated to fit the range -32000 to -32099

Due to shortcomings in a third-party crate, we're not able to distinguish between an error due to the request containing invalid JSON, or the request containing valid JSON, but not a valid JSON-RPC request. As such our server returns -32700, "Parse error" for both cases.

#848 EE-1163: Native transfer error charge

• This PR will now use a precomputed finalization result for user errors (https://github.com/CasperLabs/casperlabs-node/blob/4ad2e250ac839c41c9761e1aaba3281b17e34bb2/execution_engine/src/core/engine_state/execution_result.rs#L254-L275) in native transfer instead of early exit with 0 cost.

• During refactor I noticed I used wrong field for charging cost of wasmless transfer

• I also verified other test correctness in the meantime by changing default gas price to >1 and corrected failing tests.

#847 HWY-238: Highway synchronizer unit test

This removes all Timestamp::now() calls from the synchronizer and adds a unit test that doesn't use the wall clock. The test purges vertices from all three queues, and in the process also exercises pushing and popping vertices, and enqueueing vertices with a future timestamp.

#846 NDRS-799: Introducing effect for missing_trie_keys

#845 NDRS-839: Make consensus component `Send` by adding trait bounds and swapping `Arc` in for `Rc`

#844 NDRS-587 update readmes for crates

Updating README.md for all crates to point to casper-node repo and removing all relative links.

#843 NDRS-797: allow node to be upgraded

This PR completes the initial requirements to allow a node to be upgraded. It is comprised of 3 commits as follows:

• 7a7dd02

  is a major refactor of the chainspec structure and the chainspec loader. The notion of upgrade points is removed, with the intent that every new version will be installed with a complete chainspec. This is the simplest way to handle supporting nodes upgrading along with nodes being installed for the first time at a version > 1.0.0. Note that these changes also include the requirement that versions be installed to a folder named e.g. 1_0_0, and that the node's config.toml, the chainspec.toml, and the accounts.csv (if provided) all be installed to such a folder.

• eb8e2fc

  adds the functionality to the chainspec loader to query the config dir for a newer installed version, and retrieve the activation point (era ID at which the new version will begin) for that new chainspec. Note that this is a slight deviation from the original meaning of the activation point, which used to identify the last era before a new version was begun. The new meaning allows us to have an initial chainspec with an activation point of 0, avoiding special-casing the first version.

• f600789

  adds functionality to consensus which periodically (on every finalised block) calls on the chainspec loader to provide it with the next activation point if available. Once consensus deactivates the era immediately before the activation point, it signals to the reactor that the node should gracefully exit. This graceful exit (the only way in which the validator node can terminate with exit code 0) allows the casper-node-launcher to start running the newly-installed casper-node binary.

This should be sufficient for an initial version. Before the next version, we need to provide additional/amended functionality including:

• avoid calling ContractRuntime::commit_genesis() on startup

• call ContractRuntime::upgrade() as required

• identify and update sites using hard-coded protocol versions to use the current chainspec value

• implement config-migration if required (the subcommand exists, but is currently a no-op)

• implement data-migration if required (the subcommand exists, but is currently a no-op)

The PR includes a reasonable amount of unit tests, but no high-level tests. I have done some ad-hoc, local testing using nctl to ensure that nodes correctly identify a newly-installed chainspec at a higher version, and gracefully shutdown at the specified activation point.

#842 HWY-239: Pause if too much stake is offline

This prevents the remaining validators from creating more and more units (increasing the protocol state size, and thus memory usage) while too many validators are offline to finalize anything. E.g. if there's a 34% fault tolerance threshold, we need more than 67% = 50% + 34% / 2 online for liveness, so if 33% or more are offline, the other validators will pause.

During a pause, instead of units, they gossip new Ping messages, so that others can keep track of who's still online. Once enough validators are back online again, they resume.

#841 NDRS-772: Reduce number of warn events

Changed severity of the following statements from WARN to DEBUG

• "network announcement ignored"

• "finality signatures not handled in joiner reactor"

• "received finalized blocks out of order; queueing"

• "received request before finalization announcement"

#840 NO-TICKET: update node readme

This removes the last instances of make build-system-contracts from our docs.

#839 NDRS-827: Keep network identities the same through reactor stages

#838 NDRS-826: Use Switch Block Accumulated Seed For Key Block Seed

#837 NO-TICKET: fix potential panic in ScopedInstrumenter::drop

Due to being run on systems with a non-monotonically-increasing clock, the drop implementation of the ScopedInstrumenter was seen to panic on rare occasions.

This PR resolves that issue by using a checked subtraction.

It also appears to be the only instance of elapsed() - in our codebase.

#836 NDRS-447 - implement rate limiting on http endpoints using tower

This PR implements rate limiting on all our http endpoints using tower.

#835 NO_TICKET ensure transfers are included in block limited only by bloc…

This PR is intended to ensure that transfers are included in block limited only by block_max_transfer_limit.

#834 EE-1152: DelegatorNotFound while updating delegator rewards

Node while running a step function does distribute rewards using trusted validators passed from node, but between step calls, a delegator can undelegate funds from one of those trusted validators. While updating delegator rewards in a step call, a delegator can be already missing from current set of bids, but node does not know that through its trusted validators.

Original title of the ticked assumed MismatchedEraValidators error, but that was not observed during nctl testing, and arguments in the step function are assumed to match the global state information.

#833 EE-1163: Refactor `engine_state::mod.rs` file. [NFC]

This PR deduplicates some of the code between deploy and transfer methods by extracting the code to macros. It couldn't be a function b/c we need to return early in error cases so we need to "expand" the return logic on the call side.

#832 NDRS-767: Remove finality signatures from the block.

Currently when appending a finalisation signature to a block, we retrieve the block from storage, mutate it by appending the signature, then re-store it.

The relationship between a block and its proofs should be viewed as the same as that between a deploy and its execution results. A deploy, once stored, is immutable, and its execution results are stored in a separate DB (deploy_metadata_db).

The block struct should be updated to no longer support mutation. To that end, the proofs member should be removed along with fn append_proof().

Proofs should be handled by storage in a separate metadata DB. There should be the ability to retrieve a block and its proofs from storage via a single request event (c.f. StorageRequest::GetDeployAndMetadata).

The JsonBlock type can continue to hold the proofs as it's only used when providing a block to clients using the chain_get_block JSON-RPC. When creating that type, we'll make use of the new StorageRequest::GetBlockAndMetadata to retrieve all the required data in one call.

#831 NDRS-773/yamux buffer size, take two

Includes small stylistic fixes that were omitted from the original PR.

#830 EE-1140: remove grpc layer

This PR removes the legacy grpc layer. Basically mechanical, except for the concurrent_executor test deprecation.

#829 NO TICKET: joiner-dev-tmux.sh now shares assets with run-dev-tmux.sh

#828 NDRS-823: Factor out next era validators from EraSupervisor::handle_create_new_era

#827 NDRS-803: Add an integration test for a new node joining the network.

Adds an integration test that exercises a scenario where a new node is joining an already running network.

Test scenario:

1. Send a batch of Wasm transfers.

2. Send a batch of native transfers.

3. Take a note of the LFB hash.

4. Send a batch of Wasm transfers.

5. Send a batch of native transfers.

6. Start a new node and join the network using LFB hash from 3)

7. Verify that eventually, the LFB hash on the new node matches the hash on one of the old nodes.

This PR also adds new nctl command nctl-view-chain-lfb which will query nodes for the hash of the last finalized block.

Note that this requires nctl set up (as described in the nctl directory) and that there's an already running network.

#826 NO-TICKET: enable libp2p component by default

This PR reverses the behaviour of which networking component is chosen by default: now libp2p is the default, and users need to set the env var CASPER_ENABLE_LEGACY_NET to enable the old small_network component.

#825 NDRS-797: Initial work on casper-node upgrades

This PR lays the groundwork for adapting the casper-node to properly handle protocol upgrades.

The first commit adds new two new subcommands to the node: migrate-config and migrate-data. The former will be run post-install to migrate any config changes. The latter will be run by the casper-node-launcher just before running the node in normal mode to migrate stored data as required.

The second commit refactors storage to handle providing switch blocks by era ID. This is required as part of the upgrade story as a newly-started node will need to be able to determine the current list of validators and their key weights.

However, this commit is also required as part of the task which @fizyk20 is currently working on, hence this PR only covers part of the changeset required to handle node upgrades.

#824 NDRS-805: Fix multisig

• Node will correctly passes all public keys from deploy's approvals

• Old authorization tests were reviewed and compared against old CasperLabs/CasperLabs repo, and test cases were fixed where possible

• Added multisig test cases to ensure wasmless transfers are also respecting deploy thresholds

#823 Revert "NO-TICKET: upgrade libp2p deflate"

The PR should have included an upgrade for tokio to v1.1, and upgrading that breaks a lot of our code in a non-trivial way.

#822 NO-TICKET: send finality signatures to event stream

This fixes a recent regression whereby the vast majority of finality signatures were not being sent to the event stream.

#821 EE-1159: lock genesis validator bids for 90 days

This PR converts the existing era-based lock-in cliff to a timestamp-based one. Genesis validator bids are locked for 90 days.

#820 NO-TICKET: upgrade libp2p deflate

This fixes security advisory on libp2p-deflate by upgrading the version of libp2p. See libp2p/rust-libp2p#1932.

#819 Update libp2p to 0.34.0

This fixes security advisory on libp2p-deflate. See libp2p/rust-libp2p#1932.

#818 NDRS-809: Add "insufficient balance" variant

This commit improves UX so sending deploys from invalid account reports
different error than sending account that does not have minimum balance
required.

#817 HWY-226: Evict non-participating validators

This implements a slightly simplified version of the approach described in CEP 27, in that:

• only inactive validators are reported, and

• inactive validators are defined as ones that haven't created any units in an era from the perspective of the unit containing the switch block.

Also, this is only the Node part of the eviction implementation. There is also an Execution Engine part needed for this to be functional:

#816 NDRS-791: Introducing effect for putting a trie into the trie store

#815 NO-TICKET: fix JSON schema for CLValue

This fixes an issue whereby the JSON schema derived for CLValue omitted the parsed field of the JSON representation.

#814 NO TICKET: Mirroring changes from the linear_chain_sync

#813 NO TICKET: Qualifying `Error` in `effects.rs`

#812 NO-TICKET: Fix raw gas/motes conversions.

#811 EE-1160: Transfer bug

The issue here is that transferring transfer_amount - wasmless_gas_cost + X > account_balance for nonzero X returns Success and an empty list of transfers. Make sure we return InsufficientPayment.

#810 NO TICKET: Fix Reactor Macro Type Inference Bug

#809 NDRS-801: Verify that the execution matches expected during sync

The bug was introduced in #760 , where wasmless transfers got separated from the deploys but this change was not propagated when constructing a ProtoBlock. Which lead to a consensus not requesting wasmless transfers when fetching dependencies and, in a result, executing an incomplete block (hence different post-state hash).

#808 NDRS-788: Remove the ignore option from `cargo audit`

#807 NDRS-793: use jemallocator and dump queues when under memory pressure

This PR is intended to implement NDRS-793 - we will dump the contents of the message queues when memory utilization reaches an optional value set by and env var CL_MEM_DUMP_THRESHOLD_MB (in megabytes).

This is done in-process and brings in jemalloc as the global allocator, from which we extract the amount of memory that the node has allocated up to that point. We get the total memory available by bringing in the sys-info crate. This PR sets the groundwork upon which we can build out more robust memory metrics and tuning using jemalloc.

#806 NDRS-783: Track Trusted Block Header in Fast Sync

#804 NO-TICKET: Memory usage of `BlockValidator`

#803 NDRS-785: abbreviate large string JSON fields

This provides a json_pretty_print() function in casper-types which replaces large (over 100 char) strings with x chars where x is the number of chars in the field.

#802 EE-1156: prevent user alteration of mint entries

Addresses a security vulnerability in the mint logic.

#801 NDRS-770: Don't disconnect based on 'invalid' value.

The block validator needs to distinguish between the case where it has verified that a block is definitely invalid (e.g. one of its hashes is the correct hash of an invalid deploy), and the case where it just failed to download deploys, so it wasn’t able to complete the validation. The quickest workaround is to only disconnect from the sender of the “invalid” value for now, but not the senders of the dependent vertices.

#800 NO_TICKET - panic when block-proposer state is corrupted

This PR causes the node to panic if we load a corrupted state for the block-proposer. This is a short-term fix to prevent a node from starting in a bad state.

#799 EE-1153: Make a contract immutable

Presently all contracts in Casper can be upgraded, along with their internal state. This change prevents a contract from being upgraded, and by extension, immutable.

#798 NO-TICKET: Log the era supervisor's public key

#795 NO-TICKET: effectively disable libp2p message limits temporarily

Libp2p had a 1MB limit on message size, which appeared to create problems in the gossiping of larger deploys. This limit has been disabled for the time being.

#793 NDRS-765: Switch Fast-Sync with Compile Time Flag

#791 NO-TICKET: Create all parents of unit_hash_file

#790 NO-TICKET: Logging erroneous synchronizer starts

#789 NO-TICKET: Verify hashes for blocks from the wire

It's possible that the block coming over the wire has the wrong hash. This enforces that the header has the hash we requested and the block has the correct hash listed.

#786 NDRS-764: Add histogram metrics to `LinearChainSync`

#785 HWY-{209,230,231}: Remove blocking loops in the synchronizer, add timeout.

This breaks up the blocking loop that recursively adds all vertices with satisfied dependencies. Instead, only one vertex is added per event handler call, and a new event is scheduled — as a timer event with 0 delay — to add the next one. It also adds a timeout to entries in the synchronizer queues, and periodically removes entries older than the timeout. Finally, it makes the synchronizer more conservative when an invalid vertex is detected: We shouldn't remove a pending vertex just because one of its senders is faulty.

#784 NDRS-760: Track which peers had the data and prioritize them.

Track which peers we successfully have downloaded data from and prioritize them when we request the next piece of data. Every 5 successful attempts we will try 1 new peer (which is not in the set of the successful peer). If it has the data, it will be added to the set.

#783 NDRS-763: Cleanup of `linear_chain_sync.rs

Small clean-up of lienar_chain_sync.rs file:

• Extracts ReactorEventT trait to a file,

• Extracts State struct to a file.

• Wraps peer-related information with a PeersState struct and extracts it to a file.

• Adds peer to a DeploysNotFound result (follows the pattern of other results in the component).

• Adds GetDeploysResult enum variant to follow the pattern used when downloading blocks.

#782 NDRS-762: Ban peer that returns invalid data when syncing (vestigial).

Since FetchResult wasn't parameterized over peer type and LinearChainSync was working with I, it was not possible to ban NodeId type. Now that the missing piece of work is done (see #781), we can ban invalid nodes.

Based on (and blocked by) #781 .

#781 NDRS-761: Parameterize `FetchResult` with peer type.

LinearChainSync component uses FetchResult's peer but it was hardcoded to NodeId while the whole component operates with abstract type I. This is a vestigial element – everywhere else peer type is parameterized.

#780 EE-1142: Refund percentage

Previously, the system would refund unused token provided in ‘payment-amount’. This has been set to '0' (No refunds). Preventing a vector where a user could pretend to be willing to pay a higher amount, and just get a refund instead.

#778 NO TICKET - direct queues to dump to /tmp with timestamp

This PR directs the queue dump debug mechanism to /tmp because, when cwd is not set to a writable directory, we simply crash. This also appends a timestamp on the filename.

#777 NO-TICKET: CLValue to JSON

Full implementation of CLValue to JSON parser. It is used to produce parsed_to_json field when querying RPC for a CLValue.

#776 NDRS-758: don't hold ExecutableDeployItem args in serialized form

This updates the args fields of all ExecutableDeployItem variants to be of type RuntimeArgs rather than a serialized form of that.

#775 EE-1050: remove system contract wasms

Before this change, it was not possible to run a node purely compiled from source, without using distributed system wasm files. These were vestigial - hence removed.

#774 NO-TICK: Improve logging in the linear chain sync component.

#773 NDRS-708: Cache finality signatures.

Performance optimization

#772 NDRS-706: Provide JSON representations for Contract and ContractPackage

We need to provide Contract and ContractPackage structs in the json_compatibility module which equate to those in the contracts module of the casper_types crate. These structs should avoid the use of BTreeMaps and HashMaps to allow a proper JSON schema definition for them.

#771 NO-TICKET: add wasm-strip as an optional post-build step to each contract in the Makefile

CI Optimization

#770 NDRS-754: improve deploy validation error handling

This PR improves the handling of deploy validation, in particular resulting in more helpful RPC responses when a client sends an unacceptable deploy. Where currently the client simply receives a message stating invalid deploy, with these changes, responses will include details like:

#768 NDRS-751 - Fetcher for ReadTrieResults

#767 NDRS-737: Add memory metrics for the joiner

Should see memory joiner metric in metrics feed

#766 EE-1143: Use real gas price

Accept the user provided input for gas price and use that instead of a hard coded number from the chainspec.

#762 OP-1748: Support DNS resolution in new networking component

Support DNS names along with IP addresses in networking.

#760 NDRS-735 - separate deploys and transfers in BlockHeader

Provide dedicated space in the block for transfer transactions.

#758 NDRS-752: remove vestigial integration contracts

#755 HWY-229: Use the same numbers for adaptive round length as in the paper.

The paper suggests to slow down if ≤ 10 out of 40 rounds were successful, and to speed up if ≥ 32 were. This makes us a bit less eager to slow down, and a bit more eager to speed up, which is appropriate if we set the minimum round length to a reasonable value anyway.

Compared to the current code, this will make us

• less eager to slow down and

• more eager to speed up.

#754 EE-1141: Hide purse APIs from docs.

We plan to remove the purse model, start hiding purses in the documentation.