Apart from the big players in the cloud hosting industry, AWS, Google Cloud and Azure there are several key players that built their way around providing simple infrastructure that makes it easy and affordable to spin up a new virtual machine within seconds. Those include DigitalOcean, Linode, Vultr, OVHs Public Cloud and Scaleway, a subsidiary of Online.net.

Unlike the big three, those providers don't offer a lot of different infrastructure products. They started with a basic feature set, mainly compute instances (virtual machines) but some are now also offering block storage to extend the instances disk space and object storage to simply store a lot of data. We’ll look into which provider does offer what and how their products compare against each other.

In the recent days and weeks there were a lot of changes in this space. Linode announced their $5 plan and high memory plans as well as Vultr, who announced a $2,50 plan and improved the specs of their plans in general. On those two platforms you'll get more bang for your buck now. OVH and Scaleway are known for their cheap pricing anyways, you'll see below why! DigitalOcean still follows their initial pricing from 5 years ago, they now have the choice to lose market-share and keep their pricing or improve the specs of their plans as well.

The cloud is becoming more and more affordable!

Let’s start with comparing the products provided by those providers.

Now that we know which company provides what products we will take a closer look at the virtual machine pricing. Since this is the base for most projects and the most important aspect. Since those providers offer a cheap entry-plan but become more expensive non-linearly we’ll pick the 2GB RAM offering out of all those companies and compare them.

What we see here is the data provided on the pricing page. But you shouldn’t simply compare those, since the CPUs models vary from "cheap" Atom cores to regular, cloud-like E5 ones. Also Scaleway has a very weird storage system that is not local, unlike of all other companies, it’s some network storage solution that can become very very slow at times and very unpredictable. But we’ll see all of those details in the benchmarks.

What CPU model are in use at Vultr, OVH and co.?

All companies provide the core count in their pricing overview, but what CPU models are actually used? This is way more important than the actual count of threads you'll have access to. We’ve tested at least two locations of each provider to have a wider range of information.

[0] Disclaimer: It's not visible what exact model Vultr and OVH are using. OVH states on their website they are using an E5v3 with 2.4 GHz which matches the QEMU output and can only be an E5-2620 v3 or E5-2630 v3.

As we can see Scaleway offers by far the worst CPU out of all contenders, at least on paper. DigitalOcean is using an E5 L to save some of those very high power costs, the CPU should be more than enough for most tasks. Linode is offering the best CPU model while Vultr and OVH offer very good CPU models that are better than the DigitalOcean one but not quite as good as the CPU used by Linode. Please note that DigitalOcean, Linode and Scaleway are very transparent about their CPU, which is a good thing! Vultr on the other hand is trying very hard to cover up any hardware they use. Even if they are even using RAID is unclear, but that's a topic for itself.

CPU Benchmark

Since relying simply on stats is probably the worst thing you could do, we'll show a regular sysbench benchmark of all those CPUs. Since regular benchmarks are sometimes not comparable with real world usage we've also tested how well those CPUs perform at video transcoding with FFMPEG. This way we have two different stats to compare.

The host systems of those providers can be differently crowded, that means your newly created machine can have a lot of "neighbors" on the host system, while it could also be a nearly empty host system. When you have "noisy neighbors" the performance will drop signifantly, this is why we've performed the benchmark in two different locations, to get as many data points as possible.

Most providers only offer one thread with their 2 GB RAM plan so we were performing the sysbench with 1 thread and for providers with two threads we've ran the command additionally for 2 threads.

sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run

Lower is obviously better, this means Vultr wins the single thread benchmark. To our surprise the two Atom cores have beaten two E5 L CPUs, but only in Location 1! Good that we have done the benchmark in two locations ;-). We shouldn't forget that DigitalOceans' plan is twice as expensive as Linode or Vultr, so you get way better performance on those platforms. OVHs CPU performance is actually quite awesome if you think about that you are only paying $3,50 for this machine.

Next we'll perform the video transcoding job. We've chosen to transcode a 4K video into Full HD with a consistent bitrate.

ffmpeg -i download.mp4 -vf scale=1920:1080 -c:v h264 -b:v 5000k -minrate 5000k -maxrate 5000k -c:a aac -strict -2 -y -b:a 128k /opt/video_1080p.mp4

What we can see here is that those Scaleway Atoms aren't actually as fast in a real-world scenario. We shouldn't forget that DO and Scaleway have two threads compared to one thread so it's no surprise DO offers slightly better performance. I think the host system in location 1 was very crowded, because Linode, Vultr and OVH were beating DOs two threads with only one!

Bottom line: Vultr offers the best CPU performance out of all contenders, but OVHs plan is only $3,50 compared to Vultrs $10 and DigitalOceans $20 so it's a very good offer. If you think about it, you could get 5x OVHs plan for the price of 1x DigitalOceans plan.

Disk performance

Now that we’ve tested how capable the CPUs are, we are going a step further to test the disk performance. There are a lot of factors to consider for this test and a lot of factors are influencing the disk performance. We would guess local storage should be faster in most cases and offer more predictable performance than storage over the network. All providers apart from Scaleway offer local storage.

We'll use fio to test the disk performance. This is not quite a usual use-case but will give us some benchmark that is comparable among all providers, while with other benchmarks it's not really comparable. We'll have 8 Jobs with 512 MB each, that makes it 4 GB overall and double the amount of RAM, which is important. This test will be performed in two locations as well.

fio --name=randwrite --ioengine=libaio --iodepth=16 --rw=randwrite --bs=4k --direct=1 --size=512M --numjobs=8 --runtime=240 --group_reporting
fio --name=randread --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=512M --numjobs=8 --runtime=240 --group_reporting

We are really surprised how bad OVH does in this benchmark, they claim to have local RAID 10 storage, so it could be that they are throttling, since we can't imagine that there are that many "noisy neighbors" in two locations, but we can't say that for sure.

Scaleway does better than expected, but we noticed that it is, like the network, inconsistent and unpredictable. The performance is not really stable, so we don't know what to think about Scaleway. The benchmark does look good on paper though.

Other than that Vultr is offering great performance once again and we are kinda disappointed from the write performance of DigitalOcean and Linode but it is more than enough for most tasks, so we shouldn't be too picky here.

Bottom line: Vultr is the winner here as well. We don't know what's up with OVH, but even for $3,50 the performance is not really great. DigitalOcean, Linode and Scaleway are OK, but we shouldn't forget that DO is double the price now!

Network performance

We think that benchmarking the network is more than comparing some numbers, since the network is probably the most complex part. The quality of a network is dependend on a lot of things, from the actual capacity, to the transit providers, peering count and the network mix itself. Of course it's also important what network speed each machine has and this is were Vultr, DigitalOcean and Linode have stepped up quite a bit. In new locations (FRA1, SFO2) DigitalOcean offers 10 GbE for each machine, Linode now offers 1 GbE out and Vultr seems to have 10 GbE in most locations on new host systems.

OVH is limited to 100 MbE per machine, but you have no traffic limit, so you can burst the whole 100 MbE all day and night and you don't pay anything for it! Scaleway doubles the 100 MbE of OVH and offers unlimited 200 MbE.

What we really miss for all of those providers is some kind of transit map per location, where we can see what transit providers are connected in each location. SoftLayer has a very nice, transparent, overview table for exactly that. We would love to have something like that for all cloud companies, since this adds a lot of transparency.

Edit: As noted by a lot of people Scaleway and OVH do indeed offer "Network Maps", you can find them here: Online Network Map and OVH Network Map. DigitalOcean, Linode and Vultr: It would be great to get something similar from you guys as well :).

To test the network performance We’ve chosen to only test one location per provider but to test them in-depth. For DigitalOcean, Linode and Vultr we’ve selected their Frankfurt region, since it’s the closest to us and we know quite a bit about the networking quality in that area. Since OVH and Scaleway don’t offer Frankfurt locations We’ve chosen to compare their non-main-locations in Strasbourg and Amsterdam.

Since some providers offer 10 GbE uplinks it’s not easy to find proper test files, this is why we’ve tested the inbound speed with test files from Cachefly and 10 Gbps. What we've done additionally is that we’ve created two machines in each location and tested the private network performance as well as the public performance from those two machines.

Each test was performed 5x within a time frame of 5 minutes. The listed values are the average.

wget -O /dev/null http://cachefly.cachefly.net/100mb.test
wget -O /dev/null http://fra.download.10gbps.io/10000mb.bin ––––– for Scaleway AMS: wget -O /dev/null http://ams.download.10gbps.io/10000mb.bin

wget -O /dev/null private_ip

wget private_ip

wget -O /dev/null public

wget public

iperf3 -c private_ip -i 1 -t 30

iperf3 -c public_ip -i 1 -t 30

We can see that it doesn't matter what we do, OVH allows us to burst the 100 Mbps all the time. Other than that it's weird to see the Vultr private network is pretty bad and the public network is way better, if not the best once again. It's quite cool to be able to push > 5 Gbps on a $10 Vultr machine, what we even see here is that the disk is slower than the network, who would've thought that! DigitalOcean and Linode are great, because they offer a more consistent performance. Scaleway is below average and it was very weird that we were able to push > 200 MB/s at times but it always went below 80 MB/s after some seconds.

Now that we have the numbers, we should think about who and how our project gets visited. In Germany the market share for local ISPs is mostly split between DTAG and UPC, this means that providers in Frankfurt should have direct access to those networks (transit), but because DTAG and UPC are known for it's ridicously high transit costs we'll check out who pays the premium and who does not! Most germans use those two providers, so it’s important to reach those people as fast as possible.

It’s quite funny that all providers in Frankfurt don’t offer a direct connection to DTAG or UPC, they save the money and let other networks take care of it. OVH has a PoP in FRA and is currently building a data center in Germany, so it’s awesome to see that they have direct transit to both of these providers and pay the premium to get them! The network quality of Scaleway in France is pretty good, but the AMS location is not really great, since most of the traffic is routed through Paris first, which adds another 10ms latency.

Bottom line: We think it's very good to see that we can get such a great network performance out of such cheap servers. Think about how expensive it was to get a shared 10 GbE port 3 years ago, now we get them with a $5-10 machine! OVH probably offers the best network, but it's throttled at only 12 MB/s, compared to the competition that feels a bit underwhelming. Scaleways network is good, sadly inconsistent, but as with OVH you don't pay for any overusage. If you need to push a lot of public bandwidth, you should go with Vultr, otherwise OVH is a great fit. If you need predictable performance you should think about DigitalOcean or Linode.

Final Words

Fascinating to see that the performance is really good in general, but performance is not everything. In this blog post we weren't able to cover SLAs, support quality and so on. It depends completely on the workload and project what's best for you. We will split the final words by provider.

---

DigitalOcean

DigitalOcean has a great product. Slick design and panel, the platform is mature and reliable. The performance could be better compared to the competition but it is solid. The support is average, but we are glad that they have Jarland, who takes customer support very serious. We will eagerly follow what DigitalOcean will do after Linode and Vultr stepped up their game.

For us the question is if they will keep their pricing or also upgrade the RAM on their plans. They could lose a lot of market share to the competition, but on the other hand doubling the RAM will take a lot from their profits, let's wait and see what they decide to do.

Linode

We are not really sure what Linode has done in the past 5 years. They still have the old control panel, we shouldn't forget that they have lost all our data more than once and still, several years later there is the same, old control panel. They also handled those incidents not really well. Also, with the last update they simply changed the API IDs of their plans, which is a no-go for us developers in itself. It also seems Linode is just overwhelmed with everything. They are working on a new panel, but the progress seems to be slow.

One thing that bothers us is that they don't collect VAT, which is required by the EU law. DigitalOcean and Vultr do collect VAT and there is a very high fee if you don't collect it. Normally you would also have to pay it for the past, so if Linode starts to collect it now, they have to pay a lot of VAT money to the EU.

It's great to see they introduced the $5 plan, and improved their specs. We won't talk about the DDoS attacks, because we think this could happen to everyone, even though you should think about preventation before hand and not while you are under attack. The support is good, friendly and helpful and we think the performance is more than acceptable for the price. Will we trust Linode with our infrastructure needs though?

Vultr

Vultr really surprised us. We always had a negative feeling about them, because they are not really transparent and the panel sucked in the early days. They don't disclose any information about their hardware and it's not even certain that they use RAID since they won't comment. Other than that, you can see that they have taken great inspiration from the DigitalOcean design team but have integrated their own style, which looks great. The performance itself is good, the control panel itself has some quirks but it's useable.

There are some great features that are unique to Vultr, like the possibility of a free BGP session. We havn't had the greatest experience with their support team in the past but that might have changed. When we were trying to create the machines for the benchmarks we were greeted by a message that they blocked our office IP and we couldn't do anything! After writing them, their support fixed it within 2 minutes, we are still not really sure why this happened...

OVH

If you don't need support or any kind of SLA we think OVH is a solid choice for the price. They are expanding fast and are currently building three new datacenters (real, physical datacenters) at the same time in Germany, the US and the UK.

They have had huge problems with their regular cloud instances though (not the local storage ones), they are based on Ceph and were from time to time unuseable. Also it's sometimes a gamble if the system creates your machine or if it will get stucked, we also had the problem that we couldn't delete instances, they somehow where stuck. Their whole system is based on OpenStack and it seems they have problems managing it (who doesn't).

Speaking about the local storage instances... The CPU performance is awesome for the price, but the Disk performance is bad. We would also very much welcome a network bump to 1 Gbps, or at least as an upgradable option.

Scaleway

We like the idea behind Scaleway, they offer a "bare metal cloud" with predictable (CPU) performance. At least that's the plan, but the storage system is so weird and inconsistent. Max. amount of storage per block is 150 GB (why?), so even if your instance on the pricing page has "200 GB SSD Disk", it's splitted in two volumes, one with 150 GB and one with 50 GB. For a few days now they offer the new "Workload Intensive Cloud Servers", with Xeon D CPUs and a great price performance ratio, We think they have potential so it would be great to have them starting at 5 EUR / month.

Scaleway would be really great if they'd offer local storage or at least some kind of better storage solution. Also it's not quite visible if the storage is in RAID, so better be prepared for data loss (which you should be anyways). It would also be great if the network in Haarlem would be not routed through Paris, because that's just an additional 10ms latency. Another thing is the object storage solution, apparently it's there, but you can't use it since they are out of stock for years now.

It would be great to have some kind of private network between your dedicated servers from Online as well as the Scaleway instances. OVH is offering something similar with the vRack.

We'd like to know what your experience with those providers are and you are very welcome to comment if you have any questions or something is not clear enough.