Steem Blockchain Update August 2017

The last few months have been busy for our blockchain team. We are making great progress and want to share what we have been doing with you.

HF 20
A little over a month ago we made the tough decision to postpone HF 20 implementation in favor of enhancing our application framework. We discussed this decision with the witnesses and they were supportive of this change. We are still planning on implementing and releasing HF 20 in the coming months.

Scaling the Blockchain
In the meantime, we have been working on some fundamental changes to help scale Steem. We currently run 7 steemd nodes in AWS to support steemit.com and will need to continue to run more and more to support our growth. The key problem is that Steemd is single threaded, however, we are well along in the process of making it multithreaded.

Imagine the steemd server program is a stove. Right now, it has a single burner and we need to make a meal. A few of the dishes are the peer to peer network, accepting and evaluating blocks, and responding to API requests for data clients like steemit.com. We are rapidly switching pots, pans, and skillets onto the single burner to make each component of the meal. The increased request volume from steemit.com is delaying the meal.

Thus far, we have been buying extra single burner stoves to distribute the load because this was how blockchains were originally designed. But this is not ideal because all of our stoves (our computers) actually have 8 burners (CPU cores) and we are only using 1 each. We have been making changes to the steemd server program so that we can use all 8 burners and not need as many stoves. That way we can put a skillet on one burner, a pot on another, pan on another, you get the idea.

Previously, we had a single application that handled the p2p code, the database, plugins, and APIs. Now, everything is a plugin and only needs to communicate with other plugins that they directly depend on. This increased level of modularity allows for quicker development, easier code review, and greater parallelism. It will also allows us to switch components in and out without breaking the greater whole. We are now developing with building blocks rather than specifically designed puzzle pieces.

FABRIC
Our early tests are showing a 5-fold increase in throughput for servicing API requests. This refactor also lays the foundation to scale steemd into the future and implement the Steem Blockchain Fabric.

There are some API changes on the steemd node that should make accessing data in the blockchain easier and allow upgrading of our APIs without breaking existing calls. We have added a condenser_api which exposes all existing APIs so that we do not break backwards compatibility with existing apps. We have also rewritten the jsonrpc parser and it is completely jsonrpc 2.0 compliant. We will announce when this is deployed to staging.

Most of the work we have been doing is highly technical and challenging, but we look forward to keeping the community abreast of our work and strategies, on much of which we look to the Witnesses and community for collaboration. We’re happy to announce that we are confident these solutions will provide a robust foundation for meeting all of our future scaling needs and enable us to turn our attention to the features you all want to see.