Developing apps for block chains

Oops. I said I wasn’t going to blog about Bitcoin for a while, but Coinbase recently took an official position stating that Ethereum is now at the forefront of the digital currency space. I can only agree with 99% of Fred’s writing: his analysis of the problems of Bitcoin’s developer community mirrors my own.
But I noticed a small misconception in Fred’s post that I wanted to quickly correct, and that’ll lead on nicely to a discussion of writing blockchain apps. He said:
Bitcoin’s scripting language is intentionally restrictive. You might liken it to programming with an advanced graphing calculator — functionality is limited. As a result, you can only do basic things. It is also hard to understand and use. Rather than most modern programming languages where the code is almost readable like a sentence, it looks like unintelligible machine code. As a result, it took Mike Hearn, a talented ex-Google developer, a whopping 8 months to write a first version of a fairly simple crowdfunding application.
Fred is talking about Lighthouse, a P2P Bitcoin Kickstarter equivalent that uses the block chain:

It’s true that Bitcoin’s so-called “Script” is an assembly language rather than a scripting language as the name implies. But that isn’t why it took eight months to write Lighthouse. It would have taken nearly as long even if it was built on top of Ethereum because the way Lighthouse uses scripting is trivial. Working with Bitcoin script was probably only about 2% of the total effort. If we as an industry misunderstand the pain points in developing decentralised financial applications then we won’t be able to fix them.
And in case you’re wondering, yes I’m still working on blockchain-type technology, this time at R3 on Corda, announced in this blog post, so I’m still thinking about these things. Many of the issues Bitcoin and Ethereum developers face are the same even on a private, institutional distributed ledger system.
Sidenote: you can read a white paper written by Vitalik Buterin (or an executive summary) on Ethereum in institutional contexts that was commissioned by R3 CEV.
Corda is an attempt to build a blockchain-inspired system for industrial rather than consumer use cases: we consider our “hello world” type app to be something like an interest rate swap, rather than a crowdfund or token issuance. The applications institutional customers want to use blockchains for are far more complex than anything done on top of Ethereum/Bitcoin so far, and so when we started designing Corda I knew we’d need to find ways to radically improve developer productivity. There are just so many time sinks when writing apps for a block chain. Corda doesn’t use proof of work, but it does use a similar transaction and ledger model to Bitcoin. So if making a simple app for Corda took as much work as making a simple app for Bitcoin does, we’d have a serious problem.
I’ve built quite a few p2p blockchain apps. Not just Lighthouse but also smaller things like PayFile (micropayments for file downloads), a non-public micropayments-for-wifi Android app, and an app to do document timestamping on the block chain. Bitcoin apps don’t have to take eight months: that last example was written from scratch in a 30 minute live coding tutorial at DevCore 2015.

By now I suspect I’ve written more actually usable decentralised apps on top of the block chain than anyone else, if we define “usable” as having a GUI you could reasonably give a non-developer friend, and “decentralised” as meaning the original website can disappear but the existing users don’t notice.
The design of Corda is heavily informed by those experiences.
Here were the top time sinks when I wrote Lighthouse and what we’re doing to fix them.
Building the wrapper app
Even for an apparently simple app there were many parts to Lighthouse that mattered but are not obvious, like the embedded wallet. That implied features like backup/restore using the 12 random words scheme, QR code rendering and so on. The bitcoinj library provides a “wallet template” app that you can copy as the basis for your own Lighthouse-like program, but copy/paste is a poor approach to code reuse.
Ethereum is going in the right direction with Mist, a “dapp browser”. But Mist is really meant for developers, not ordinary users. Some operations expect you to copy/paste JSON directly into the UI, for instance. And you have to make your UI in HTML/JS, a platform which has poor productivity and is also difficult to secure.
The long term plan was for Lighthouse to evolve into an app platform like what Mist is aiming to be, so people could write downloadable sandboxed apps that’d be fetched from a kind of blockchain app store, and they wouldn’t have to worry about these details. Lighthouse is not built on HTML5 so doesn’t suffer the inherent security issues of that technology.
In Corda, a “cordapp” is a collection of smart contracts that verify transactions, protocols to build and do joint transaction signing, API plugins for the server and perhaps eventually also UI plugins. These can all be sandboxed and loaded dynamically. The APIs are a defined part of the platform. In this way, generic services useful for all such apps don’t have to included as a library, rather, the app is a library loaded by the node software itself.
Fees
It is typical for blockchain platforms to insist that every transaction has a fee. In Bitcoin, this is true even if it has so much coin age it’s obviously not a denial-of-service attempt. It’s amazing how much complexity this adds to what would otherwise be simple code.
In Corda there is no native crypto-currency and thus no transaction fees. If one day the network becomes open to the entire world it would be structured like the internet itself, where end users gain access to the core network through providers that take responsibility for abuse handling, rather than the core network charging some arbitrary fee per packet and hoping that’s good enough. Contracts have hard-coded opcode/memory quotas set high enough that it should never be a problem unless a contract goes into an infinite loop.
The P2P network
Peer to peer systems involve simultaneously talking to lots of ‘servers’ that can die, move around, respond slowly or not at all, and provide incorrect information. This is a very different model to the standard web app design where there’s a single trusted server and the client/JS developer just assumes the server is fast, online and trustworthy. It complicates the client tremendously and often involves difficult tradeoffs.
I had the benefit of building on top of bitcoinj, which handles a lot of these complexities for you, but it only got me 90% of the way.
Bitcoin provides a single hammer, the block chain. And it is a mighty hammer. But even if you had Thor’s hammer building an entire house with it would be painful.
Often when building apps you want point-to-point transient communication between users, not just global broadcasts that are recorded forever. In Lighthouse this is provided by a coordination server, or if you don’t want to use one of those you can move data around using email/slack/any kind of file transfer. It’s useful because crowdfunding projects and pledges include things like images, rich text, personal messages and other stuff that doesn’t make sense to drop into a financial ledger. But it means having to write separate servers, protocols, discovery logic and client code.
The lack of any infrastructure for solving this problem routinely leads to bizarre hacks, like creating transactions that don’t actually change anything on the ledger and are there just to carry encrypted data between two users. It wastes resources for everyone, increases fragility, and complicates the programming model. And the Bitcoin network imposes lots of arbitrary limits to try and stop people abusing the blockchain in this way (but, of course, doesn’t provide any convenient alternative).
The need for a store and forward network was something I wrote about back in 2014. But no such network exists. I think Ethereum has the beginnings of one in Whisper, but it’s not clear to me how mature it is.
Corda is based on standard messaging queuing protocols (think: email for machines). Nodes are assumed to be semi-stable and have a long term identity, so sending a message to an identity that is offline simply queues it up for later delivery. Because it builds on standard protocols there are high quality, high performance implementations available that manage the details of making this reliable. You can attach arbitrary files to transactions that are then available to contract code, and those attachments are streamed over the network automatically and efficiently.
Additionally, Corda has no reliance on global broadcast. Consensus mechanisms are pluggable as long as they provide finality. Thus, there are never any cases where multiple peers are giving you conflicting information about the state of the world. You never need to download or process an entire block chain. You don’t have to handle transactions being reversed. All these things dramatically simplify your code.
Lifecycle management
A kickstarter style crowdfund has a fairly simple lifecycle, but making sure it’s tracked and state transitions are correctly enforced is still complex. This is especially true when you recall that peer to peer networks are not always consistent and the block chain does not provide finality, so transactions can be rolled back or invalidated.
Worse, the state the UI is tracking can be updated by other instances of the app at any time, which means you have to elegantly handle race conditions like “I am attempting to pledge money to a project but whilst I was sitting on the pledge screen the project became closed to new pledges” or “another app spent my money out from underneath me whilst I was in the process of trying to create a transaction”. You’d better be good at handling concurrency, otherwise you’re gonna crash in obscure situations that you probably didn’t test well.
Oh, and don’t forget to checkpoint everything to disk at the right times, so you can come back from arbitrary power outages and app crashes.
Most blockchain apps simply ignore these kind of details because such cases are rare, and so it’s easy to skip them without anyone noticing. But handling it all takes a lot of fiddly work.
In Corda we are prototyping a framework that automates most of the tedious, error prone work in this area. R3 CTO Richard Brown was surprised when this was one of the first things I started implementing, but given my experience writing Bitcoin apps it was an obvious place to begin. Asset and trade lifecycles can be implemented as straightforward, linear code in ordinary programming languages. You can use loops, branches, and so on. We call these things protocols. Behind the scenes the control flow graph is transformed into an automatically checkpointing state machine. It is reasonable to block for days or weeks waiting for something inside a protocol. Protocols can interact with abstract institutional identities, specific people, and internal behind-the-firewall systems. The network layer figures out which IP addresses currently corresponds to an identity using an advanced network map. This is especially vital because so many of the things Corda will be used for are inherently multi-party.
Testing
Too many people skimp on this. Buggy contracts have been identified by Vitalik as one of the top issues facing Ethereum. Corda has the beginnings of a DSL-like API which makes writing contract unit tests easier. It also has an early version of a tool that performs and visualises network simulations, one thread per node with simulated latency, which can be seen operating in this presentation by Barclays.

Learning new developer toolchains
It is conventional for existing blockchain systems to invent their own languages and bytecode sets for expressing smart contracts.
Corda smart contracts are Turing complete, but we think it doesn’t make sense to invent a new developer toolchain from scratch. Contracts are not so different from ordinary apps that you need to write a whole new IDE — this is extravagance. I see that Ethereum is thinking about switching to WebAssembly, so maybe they have reached a similar conclusion. Corda adapts an existing industrial strength toolchain that developers are already familiar with, meaning a top quality IDE and debugging experience right from the start. Still … no specific language is mandated, so if someone did invent a compelling custom language, it could be used.
Time management
A frequent Lighthouse feature request (never implemented) was to automatically invalidate all pledges when a project reached a deadline. Bitcoin does have a way to interact with time: a feature called nLockTime, but actually making things happen once the lock time was reached would have required more manual effort again.
Corda has a basic event scheduler API that interacts with its equivalent of the wallet, making it much easier to write apps that do things when certain times are reached.
Misc
Offline support, error handling, forking Bitcoin Core to add a performance feature the upstream developers rejected (that was an extra 2 months of work on Lighthouse right there), a users guide, proper code-signed downloads, creating a threshold signature hardware-protected online update scheme, etcetera etcetera. It all adds up. Those 8 months went by pretty fast.
Conclusion
Whatever block chain platform you’re interested in, it’s important that we all understand what makes developing these new kinds of apps difficult. Whilst a more powerful scripting language can help a lot, that’s by no means the only thing that needs to be improved.
I’ve outlined a few of the ways the Corda project is tackling developer productivity. These are not all of the design ideas we’re exploring, just the ones that are most relevant to this article.
Did you think it sounded interesting? If you’re in the London area and would like to take part then check out our jobs site. If you’re not in London but have got a ton of blockchain experience and enthusiasm, then get in touch anyway: I’d still like to have a chat with you.