Dogfooding and the journey towards Populus 1.0

I'm currently towards the very end of a very large refactor of the Populus development framework and I thought it might be interesting to shed some light on how I write software.

Populus was started shortly after the launch of the Frontier network to fill a personal need.  As I tinkered and fiddled with the Ethereum network I found two major areas I was frustrated with.

First, developing smart contracts was tedious.  I would write a contract, use the solc compiler to generate the bytecode, fire up the geth console, do a bunch of copy/pasting to generate a deployment transaction, do a bunch of copy/pasting to get the contract ABI loaded into a web3 contract object, etc.  It was very manual and tedious.

Second, and probably most important was that I had no way to write automated tests for my code.  One of my personal developer soapbox topics is testing, and I was very frustrated with my inability to iterate quickly and still know that my code worked as expected.

Out of the void came Populus.  The name Populus comes from the genus that the Aspen tree belongs to.  Aspen trees are interesting in that their root systems are interconnected, much the way that many of us in the Ethereum community envision the future of smart contracts.

Over the coming months the feature set within Populus would grow organically to fit whatever needs encountered while developing the Ethereum Alarm Clock service.  For anyone who has some depth of experience in writing software, this sort of unplanned organic growth is an execellent way to acquire a healthy dose of technical debt.

The current rewrite really began in February of this year.  I had begun extracting pieces of functionality that originated within the Populus codebase into their own libraries and began converting the main Populus codebase to make use of those libraries.  Unfortunately, around the same time, my amount of free time available to develop the framework all but disappeared and the effort stalled.

Fast forward to June of 2016.  After working with the Ethereum Foundation on the Casper proof of concepts I found myself in an ideal position to help introduce python 3 support to the pyethereum codebase.  Around the same time, I started trying to pick up where I left off months before and found that the majority of my tooling was fundamentally broken.  Many of Populus' dependencies had changed sufficiently that the framework was largely broken unless you wanted to use very old versions of the underlying software.

For the last 3 months I've been working on an updated python toolchain for Ethereum development.  Along the way all of the following tools have been either created new or fully updated, each laying a piece of the foundation I needed to get the Populus codebase to a place I could be happy with.

• py-solc: solidity compilation from python
• py-geth: management of running a go-ethereum node from python
• eth-abi-utils: ABI encoding and decoding
• web3.py: Simple and easy backend agnostic blockchain interactions
• eth-testrpc: Fast in-memory EVM interactions backed by pyethereum.tester.

The first beta release for Populus 1.0 was published on July 27th, almost a month an a half ago.  The library has been almost entirely rewritten and I knew that it was both better and worse.  It was better in that it now sat on a solid foundation of well tested tools with well constrained responsibilities.  It was worse because it had been rewritten in a vacuum.  I reworked almost every piece of functionality but I wasn't actually using the tool so I had no way to measure if what I'd written was right.

One of the worst ways to write software is to focus entirely on features without thought to the way people actually use them.  I knew that the freshly rewritten Populus likely suffered from this problem and the only way to fix it was to use it and find out what parts were hard to use.  As it turns out, the first beta release had a terrible user experience.

Over the last two months I've been practicing a development practice often referred to as dogfooding.  I've started stepping back into development of the Ethereum Alarm Clock service.  Each day as I work on updating the test suite I've found various warts and deficiencies in the new Populus and slowly, the 1.0 codebase has become something I feel is almost ready for public consumption.

As of today you can try Release Candidate 3 with pip install populus==1.0.0rc3.  I expect to put out RC4 later today.  Populus is very close to a stable 1.0 release, but I don't plan on doing so until I'm confident it is ready for public consumption.  I do hope you'll give the framework a try.  It has become a cornerstone in my Ethereum development toolchain.  Feel free to reach out to me in the Populus gitter channel if you have any issues or feedback.