The approach for testing distributed, decentralized systems, aka testing blockchain bears no resemblance to traditional network testing; this is a new testing paradigm from regular, client-server application testing. In conventional performance testing, you can assume control over the network and makeup throughput numbers. Blockchains, on the other hand, must understand they are working in an adversarial environment. Given their distributed nature, these peer to peer networks or public blockchains only reach ultimate finality and may go through a reorganization of their transactions. They rely on proof of work or proof of stake to ensure they reach consensus, at the cost of slowing down the throughput of the system. This is why testing the performance of the blockchain and how it reaches a consensus between the different actors is critical. Additionally, all of the infrastructures that are built on top of these networks can negatively affect performance. Oracles are also examples of the type of foundation that can act as performance bottlenecks. Simply, these act as translators for information provided by an external platform. Its sole function is to supply data to blockchains, which permit the creation of smart contracts.
Testing this new technology means that there are new metrics for us to capture. Some of these include transaction size, sender and receiver address, block time, block size, or block mempool size. Block mempools are an unprocessed set of connections in an untreated set of blocks. For example, when you receive a transaction, it will sit in the mempool until the network has the ability to process them. When we apply a load, we can see that those mempools tend to increase in size and may even become a bottleneck in terms of performance. Other metrics can also help us measure the difficulty of doing cryptographic checks on every bit of information we receive. Concerning the mempool size, we can see the time it takes to validate the block header to check that the signature of the block matches what the block contains. Through our work, we’ve found that blockchains typically estimate a theoretical performance profile based on the best conditions. We try to give the maintainers a better idea of how the chain behaves in a more realistic live environment by collecting and analyzing these unique metrics.
To collect the previously mentioned metrics, we use logs, block data, and in some cases, Prometheus. Using these tools, we gather the raw data for a test run and then leverage data science tools like Jupyter, Grafana, Elastic Search to process and make sense of it.
Our Method and Process for testing blockchain
Whiteblock tests start with the establishment of a control group, which will represent the upper bound of the performance profile. We apply no limits to networking constraints to get the best numbers possible.
Latency, Bandwidth, & Packet Loss
We then play with latency, bandwidth, and packet loss separately to estimate how these different degradations affect the chain. While latency should slow down the chain linearly, bandwidth may particularly impact noisy network protocols, and packet loss is particularly impactful to gossip-based protocols that have to perform expensive retries.
We combine those three factors together into a stress test to check the behavior of the chain under real Internet traffic conditions.
We push deeper into the networking aspect of blockchain by toying with topology testing. In the blockchain, it is essential to have the dissemination of peers and connections to avoid creating bottlenecks. We test static topologies, such as full meshed, pseudo-randomly connected, or shaped to perform a specific bottleneck to understand the throughput requirements of the chain.
We also test to ensure that peer discovery is producing valid peer organizations, without islands or bottlenecks.
When it comes to data, there are two main camps (and many new takes) on state management of a chain: UTXO (Unspent Transaction Output) and account-based. Both have different approaches to storing and reconciling data. UTXOs, for example, do not require further consolidation of the data. Account-based chains, on the other hand, need to update their account balance every time an account is touched. The world state of the chain (all of the accounts) is organized into a tree that represents the cryptographic proof of its validity.
Blockchain data is as such a weird animal, and we test its unique capabilities by bringing up new nodes on the network and examining its ability to sync. We also investigate how hard forks change the execution rules of the transactions, and therefore change the way the chain operates across its different implementations. In some cases, subtle bugs may occur that break the network and create two chains unable to reconcile consensus.
Finally, with the help of tools such as NeoLoad, we perform extensive load testing of clients. Each chain is different in regards to what it uses as transaction input, and that creates a challenge to address early on. We also like to separate load testing a particular node on the network to test its capability to offload transactions to its peers, versus overloading the network entirely by testing all the nodes present. This is done to estimate the capacity and throughput of the combination of the nodes.
Our company, Whiteblock, has executed this type of testing, stressing the network, and consensus mechanisms of blockchain systems for over 20 different blockchain protocols. Based on this testing, we delivered several reports; some of them made the public with our customer’s authorization. We are concentrated on the endless innovation of the space, and we keep coming across more requirements, such as moving more and more towards anonymized infrastructure based on Tor, or relying on UDP for peer gossip. We’d love to tell you more about it. Contact us by visiting www.whiteblock.io, following us on Twitter @whiteblock, or joining our Telegram community.
Learn More about the Performance Advisory Council
Want to see the full conversation, check out the presentation here.