benchmarking the cloud

RSS Feed

Cloud Articles From Our Blog

06/28/2010 06:11 PDT

Cloud Server Benchmarking Part 5: Encoding & Encryption

This is the fifth post in our series on cloud server performance benchmarking. In this post, we'll look at encoding and encryption performance using a compilation of 7 different benchmarks.

Benchmark Setup
All benchmarked cloud servers were configured almost identically using CentOS 64-bit (or 32-bit in the case of EC2 m1.small, c1.medium, Gandi, and IBM cloud servers).

Benchmark Methodology
Individual benchmark scores are calculated using the Phoronix Test Suite. Phoronix runs each test 3 times or until the standard deviation between each execution is less than 3.5% to improve statistical accuracy.

We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is the NewServers Jumbo server configured with dual Intel E5504 quad core 2.00 GHz processors and 48GB DDR3 ECC ram. We chose NewServers because they are the only IaaS cloud that does not utilize a virtualization layer that could adversely affect the benchmark results. All NewServers servers run on top of physical hardware. We assigned the baseline server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 7 benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Benchmarks
The following benchmarks were used to calculate the aggregate encoding performance (Encode) score displayed in the results tables below.
  • Monkey Audio Encoding [weight=100]: This test times how long it takes to encode a sample WAV file to APE format.
  • WAV To FLAC [weight=100]: This test times how long it takes to encode a sample WAV file to FLAC format.
  • WAV To MP3 [weight=100]: LAME is an MP3 encoder licensed under the LGPL. This test measures the time required to encode a WAV file to MP3 format.
  • WAV To Ogg [weight=100]: This test times how long it takes to encode a sample WAV file to Ogg format.
  • WAV To WavPack [weight=100]: This test times how long it takes to encode a sample WAV file to WavPack format.
  • FFmpeg AVI to NTSC VCD [weight=100]: This test uses FFmpeg for testing the systems audio/video encoding performance.
  • GnuPG [weight=100]: This test times how long it takes to encrypt a 2GB file using GnuPG of a file.
Results
The results are divided into tables separated by provider. If the provider has more than one data center location, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), and the aggregate baseline relative score (as described above) linked to the complete Phoronix results. On the high end, there wasn't as much performance variation between providers in this post when compared to the previous 4 performance posts. Additionally, the results show that these benchmarks appear to be most influenced by CPU model and clock speed versus number of CPUs/cores. The top performers in this post are NewServers, GoGrid and Bluelock.


Amazon Web Services (AWS) [US East]
ID CPU Memory Encode
us-east.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.32
us-east.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 136.12
us-east.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 135.81
us-east.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.77
us-east.m1.xlarge Xeon E5430 2.66 [4 processors, 4 cores] 15 103.33
us-east.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 103.06
us-east.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 101.56
us-east.m1.small Opteron 2218 HE 2.60 [1 processor, 1 core] 1.7 43.25





Amazon Web Services (AWS) [US West]
ID CPU Memory Encode
us-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.29
us-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 136.27
us-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 135.86
us-west.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.77
us-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 103.9
us-west.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 103.28
us-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 100.89
us-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 45.2





Amazon Web Services (AWS) [EU West]
ID CPU Memory Encode
eu-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 136.4
eu-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.21
eu-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 136.12
eu-west.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.69
eu-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 103.83
eu-west.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 103.79
eu-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 101.28
eu-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 44.92





Amazon Web Services (AWS) [APAC]
ID CPU Memory Encode
apac.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 136.12
apac.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 135.83
apac.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 135.49
apac.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 119.54
apac.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 103.8
apac.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 103.68
apac.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 101.32
apac.m1.small Xeon E5430 2.67 [1 processor, 1 core] 1.7 45.45





Rackspace Cloud [Dallas]
ID CPU Memory Encode
16gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 111.2
2gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 108.98
1gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 108.25
4gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 106.02





Rackspace Cloud [Chicago]
ID CPU Memory Encode
16gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 112.44
8gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 8 112.44
4gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 111.58
1gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 108.5
2gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 107.61





Storm on Demand [MI, US]
ID CPU Memory Encode
48gb Xeon X5650 2.67 [1 processor, 12 cores] 45.9 135.59
i5-750-4gb Core i5 750 2.67 [1 processor, 4 cores] 4 130.83
4gb Core i5 750 2.67 [1 processor, 2 cores] 3.5 130
i5-750-2gb Core i5 750 2.67 [1 processor, 4 cores] 2 128.51
2gb Core 2 Quad Q9400 2.66 [1 processor, 1 core] 1.7 124.27
8gb Xeon X3440 2.53 [1 processor, 4 cores] 7 123.64
x3440-8gb Xeon X3440 2.53 [1 processor, 8 cores] 8 123.4
16gb Opteron 2378 2.40 [2 processors, 4 cores] 15.2 121.05
32gb Opteron 2378 2.40 [4 processors, 8 cores] 30.4 120.76
e5506x2-4gb Xeon E5506 2.13 [1 processor, 8 cores] 8 108.32
e5506x2-4gb Xeon E5506 2.13 [1 processor, 8 cores] 8 108.31
e5506x2-8gb Xeon E5506 2.13 [1 processor, 8 cores] 8 108.1
e5506x2-8gb Xeon E5506 2.13 [1 processor, 8 cores] 8 107.75
amd2350x2-32gb Opteron 2350 2.00 [4 processors, 8 cores] 32 95.23





GoGrid [US West]
ID CPU Memory Encode
8gb Xeon E5450 2.99 [1 processor, 6 cores] 8 146.51
4gb Xeon E5520 2.27 [1 processor, 4 cores] 4 108.36
1gb Xeon E5520 2.27 [1 processor, 1 core] 1 104.3
2gb Xeon E5520 2.27 [1 processor, 2 cores] 2 103.78





Voxel [New York]
ID CPU Memory Encode
4gb-ny Xeon L5520 2.26 [2 processors, 2 cores] 4 119.95
8gb-ny Xeon L5520 2.26 [4 processors, 4 cores] 8 118.91
14gb-ny Xeon L5520 2.26 [7 processors, 7 cores] 14 118.79
2gb-ny Xeon L5520 2.26 [1 processor, 1 core] 2 106.67





Voxel [Amsterdam]
ID CPU Memory Encode
8gb-nl Xeon L5520 2.26 [4 processors, 4 cores] 8 122.62
14gb-nl Xeon L5520 2.26 [7 processors, 7 cores] 14 122.6
4gb-nl Xeon L5520 2.26 [2 processors, 2 cores] 4 122.58
2gb-nl Xeon L5520 2.26 [1 processor, 1 core] 2 114.22





Voxel [Singapore]
ID CPU Memory Encode
14gb-sg Xeon L5520 2.26 [7 processors, 7 cores] 14 115.9
8gb-sg Xeon L5520 2.26 [4 processors, 4 cores] 8 115.83
4gb-sg Xeon L5520 2.26 [2 processors, 2 cores] 4 115.8
2gb-sg Xeon L5520 2.26 [1 processor, 1 core] 2 112.29





NewServers [Miami]
ID CPU Memory Encode
fast Xeon E5450 2.99 [1 processor, 4 cores] 4 152.43
large Xeon E5405 2.00 [1 processor, 4 cores] 4 101.06
jumbo Xeon E5504 2.00 [2 processors, 8 cores] 48 100
med Xeon 3.20 [1 processor, 2 cores] 2 94.28
small Xeon 2.80 [1 processor, 1 core] 1 81.55





Linode VPS Hosting [Atlanta]
ID CPU Memory Encode
5760-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 5.54 115.71
14400-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 14.06 115.66
8640-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 8.45 108.25
1080-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 1.05 108.05
2880-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 2.81 106.65





SoftLayer [Dallas]
ID CPU Memory Encode
1gb-dallas Xeon X3460 2.80 [1 processor, 1 core] 1 126.76
4gb-dallas Xeon E5520 2.27 [4 processors, 4 cores] 4 111.51
8gb-dallas Xeon E5520 2.27 [8 processors, 8 cores] 8 111.28
2gb-dallas Xeon X3460 2.80 [2 processors, 2 cores] 2 96.74





SoftLayer [WDC]
ID CPU Memory Encode
4gb-wdc Xeon X3460 2.80 [4 processors, 4 cores] 4 137.71
1gb-wdc Xeon X3460 2.80 [1 processor, 1 core] 1 131.49
2gb-wdc Xeon X3460 2.80 [2 processors, 2 cores] 2 129.06
8gb-wdc Xeon E5520 2.27 [8 processors, 8 cores] 8 116.04





SoftLayer [Seattle]
ID CPU Memory Encode
4gb-seattle Xeon X3460 2.80 [4 processors, 4 cores] 4 131.51
1gb-seattle Xeon X3460 2.80 [1 processor, 1 core] 1 122.06
2gb-seattle Xeon X3460 2.80 [2 processors, 2 cores] 2 117.75
8gb-seattle Xeon E5520 2.27 [8 processors, 8 cores] 8 111.14





Terremark vCloud Express [Miami]
ID CPU Memory Encode
1gb Opteron 8389 2.91 [1 processor, 1 core] 1 138.13
2gb Opteron 8389 2.91 [1 processor, 1 core] 2 131.27
4gb-2vpu Opteron 8389 2.91 [1 processor, 2 cores] 4 128.26
8gb-4vpu Opteron 8389 2.91 [1 processor, 4 cores] 8 105.19
16gb-8vpu Opteron 8389 2.91 [1 processor, 8 cores] 16 95.25





VPS.NET [Atlanta]
ID CPU Memory Encode
18node-atl Xeon E5520 2.27 [7 processors, 7 cores] 6.6 106.3
12node-atl Xeon E5520 2.27 [5 processors, 5 cores] 4.4 102.46
6node-atl Xeon E5520 2.27 [3 processors, 3 cores] 2.2 94.66
3node-atl Xeon E5520 2.27 [2 processors, 2 cores] 1.1 89.67





OpSource Cloud [VA, US]
ID CPU Memory Encode
1gb Xeon X7460 2.66 [1 processor, 1 core] 1 72.27
4gb-2cpu Xeon X7460 2.66 [1 processor, 2 cores] 4 70.06
2gb Xeon X7460 2.66 [1 processor, 1 core] 2 68.77
16gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 16 63.81
8gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 8 62.54
32gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 32 61.01





Speedyrails [Quebec]
ID CPU Memory Encode
4gb Xeon E5520 2.27 [2 processors, 8 cores] 4 113.13
1gb Xeon E5520 2.27 [2 processors, 8 cores] 1 111.37
2gb Xeon E5520 2.27 [2 processors, 8 cores] 2 106.45





Zerigo [CO, US]
ID CPU Memory Encode
4gb Opteron 2374 HE 2.20 [2 processors, 4 cores] 4 110.53
2gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 109.61
1gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 108.97





ReliaCloud [MN, US]
ID CPU Memory Encode
4gb Xeon E5504 2.00 [1 processor, 4 cores] 4 38.95
8gb 8 38.81
2gb Xeon E5504 4.40 [1 processor, 2 cores] 2 35.2
1gb Xeon E5504 2.00 [1 processor, 1 core] 1 24.15





Gandi.net [France]
ID CPU Memory Encode
6gb Xeon L5640 2.27GHz 2.26 [1 processor, 6 cores] 6 125.01
4gb Opteron 8346 HE 1.79 [1 processor, 4 cores] 4 72.22
1gb Opteron 8346 HE 1.79 [1 processor, 1 core] 1 70.61
2gb Opteron 8346 HE 1.79 [1 processor, 2 cores] 2 67.37





CloudSigma [Switzerland]
ID CPU Memory Encode
8gb-8gh Six-Core Opteron 2427 2.21 [1 processor, 4 cores] 8 111.01
16gb-16gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 16 110.75
4gb-4gh Six-Core Opteron 2427 2.21 [1 processor, 2 cores] 4 110.19
32gb-20gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 32 109.59
1gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 1 95.24
2gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 2 91.15





IBM Development & Test Cloud [NY, US]
ID CPU Memory Encode
dev-large Xeon X5570 2.93 [1 processor, 8 cores] 3.5 126.89
dev-med Xeon X5570 2.93 [1 processor, 4 cores] 1.75 98.55





BlueLock [IN, US]
ID CPU Memory Encode
8gb-4cpu Xeon X5550 2.67 [1 processor, 4 cores] 8 142.31
4gb-2cpu Xeon X5550 2.67 [1 processor, 2 cores] 4 133.29
16gb-8cpu Xeon X5550 2.67 [1 processor, 8 cores] 16 130.11
2gb Xeon X5550 2.67 [1 processor, 1 core] 2 119.57
1gb Xeon X5550 2.67 [1 processor, 1 core] 1 113.3





Cloud Central [Australia]
ID CPU Memory Encode
huge Opteron 2374 HE 2.20 [4 processors, 4 cores] 16 112.23
reg Opteron 2374 HE 2.20 [4 processors, 4 cores] 4 112.1
large Opteron 2374 HE 2.20 [4 processors, 4 cores] 8 111.82
med Opteron 2374 HE 2.20 [4 processors, 4 cores] 2 103.44
small Opteron 2374 HE 2.20 [4 processors, 4 cores] 1 100.05





RimuHosting [Dallas]
ID CPU Memory Encode
tx-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 101.27





RimuHosting [Auckland]
ID CPU Memory Encode
nz-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 105.98





ElasticHosts [Peer1 UK]
ID CPU Memory Encode
8gb-20gh Xeon E5420 2.50 [1 processor, 6 cores] 8 124.54
1gb-2gh Xeon E5420 2.50 [1 processor, 1 core] 1 118.62
4gb-8gh Xeon E5420 2.50 [1 processor, 4 cores] 4 118
2gb-4gh Xeon E5420 2.50 [1 processor, 2 cores] 2 113.55





Flexiscale [UK]
ID CPU Memory Encode
4gb Dual-Core Opteron 8220 2.80 [1 processor, 2 cores] 4 126.58
8gb Opteron 8218 2.62 [1 processor, 4 cores] 8 115.72
2gb Opteron 8218 2.60 [1 processor, 1 core] 2 109.76
1gb Opteron 8218 2.60 [1 processor, 1 core] 1 105.06


06/21/2010 09:01 PDT

Cloud Server Benchmarking Part 4: Memory IO

This is the fourth post in our series on cloud server performance benchmarking. In this post, we'll examine memory IO performance using a compilation of 7 different benchmarks. This is the last of the performance posts to focus on synthetic benchmarks. Memory IO performance is particularly important for applications that read and write from memory heavily like caching systems and memory-based data stores such as memcache and redis.

Benchmark Setup
All benchmarked cloud servers were configured almost identically using CentOS 64-bit (or 32-bit in the case of EC2 m1.small, c1.medium, Gandi, and IBM cloud servers).

Benchmark Methodology
We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is the NewServers Jumbo server configured with dual Intel E5504 quad core 2.00 GHz processors and 48GB DDR3 ECC ram. We chose NewServers because they are the only IaaS cloud that does not utilize a virtualization layer that could adversely affect the benchmark results. All NewServers servers run on top of physical hardware. We assigned the baseline server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 7 benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Benchmarks
The following benchmarks were used to calculate the aggregate memory IO performance (MIOP) score displayed in the results tables below. Geekbench and Unixbench measure both CPU and memory IO performance and are included in the aggregate MIOP metric with a weight of 25 points each. To view the raw Geekbench and Unixbench scores for each of the servers in this post see the post What is an ECU? CPU Benchmarking in the Cloud.
  • CacheBench [weight=100]: This is a performance test of CacheBench, which is part of LLCbench. CacheBench is designed to test the memory and cache bandwidth performance. 50 points are assigned to each of the benchmarks read and write.
  • Geekbench [weight=25]: Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand, Geekbench takes the guesswork out of producing robust and reliable benchmark results.
  • hdparm cached reads [weight=50]: Determines the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.
  • RAMspeed [weight=100]: This benchmark tests the system memory (RAM) performance. 33 points are assigned to each of the benchmarks add, copy and scale.
  • Redis Benchmark [weight=50]: Redis is an in-memory key-value store. It includes the redis-benchmark utility that simulates SETs/GETs/INCRs/LPUSHs/LPOPs/PINGs/LRANGEs done by N clients at the same time sending M total queries (it is similar to the Apache's ab utility). Our benchmark is performed with 50 simultaneous clients performing 100000 requests (./redis-benchmark -n 100000). The result is the average requests per second for all benchmark actions (gets, sets, etc.). For specific results, download the benchmark output source.
  • Stream [weight=100]: This benchmark tests the system memory (RAM) performance. 25 points are assigned to each of the benchmarks add, copy, scale and triad.
  • Unixbench [weight=25]: UnixBench provides a basic indicator of the performance of a Unix-like system. Multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system.
Results
The results are divided into tables separated by provider. If the provider has more than one data center location, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), Redis Benchmark score (linked to the complete Redis benchmark results), and the aggregate baseline relative score (as described above). The top performers in this benchmark category were EC2, Storm on Demand (6 servers with scores 116 and above), BlueLock, and IBM Developer Cloud.


Amazon Web Services (AWS) [US East]
ID CPU Memory Redis MIOP
us-east.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 44330.61 117.88
us-east.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 53150.28 113.04
us-east.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 43767.98 102.18
us-east.m1.xlarge Xeon E5430 2.66 [4 processors, 4 cores] 15 27314.76 63.67
us-east.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 28656.14 57.05




Amazon Web Services (AWS) [US West]
ID CPU Memory Redis MIOP
us-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 42603.58 118.12
us-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 48163.06 112.11
us-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 52766.05 106.11
us-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 23420.06 70.83
us-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 31635.65 65.49
us-west.m1.large Xeon E5430 2.66 [2 processors, 2 cores] 7.5 28608.91 64.93
us-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 9214.6 31.81




Amazon Web Services (AWS) [EU West]
ID CPU Memory Redis MIOP
eu-west.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 44084.09 117.27
eu-west.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 48563.97 111.74
eu-west.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 46370.88 104.07
eu-west.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 37373.2 73.48
eu-west.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 28149.67 65.94
eu-west.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 27864.93 63.22
eu-west.m1.small Xeon E5430 2.66 [1 processor, 1 core] 1.7 9038.17 33.52




Amazon Web Services (AWS) [APAC]
ID CPU Memory Redis MIOP
apac.m2.xlarge Xeon X5550 2.67 [2 processors, 2 cores] 17.1 15859.76 97.48
apac.m2.2xlarge Xeon X5550 2.67 [4 processors, 4 cores] 34.2 15587.35 95.94
apac.m2.4xlarge Xeon X5550 2.67 [8 processors, 8 cores] 68.4 31992.3 92.74
apac.c1.medium Xeon E5410 2.33 [1 processor, 2 cores] 1.7 16492.91 68.46
apac.c1.xlarge Xeon E5410 2.33 [8 processors, 8 cores] 7 23171.72 66.47
apac.m1.large Xeon E5430 2.67 [2 processors, 2 cores] 7.5 21667.04 59.67
apac.m1.xlarge Xeon E5430 2.67 [4 processors, 4 cores] 15 18181.59 59.47
apac.m1.small Xeon E5430 2.67 [1 processor, 1 core] 1.7 7465.1 31.67




Rackspace Cloud [Dallas]
ID CPU Memory Redis MIOP
16gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 23844.89 70.09
4gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 16928.56 63.44
2gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 19167 63.04
1gb Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 15945.68 62.83




Rackspace Cloud [Chicago]
ID CPU Memory Redis MIOP
16gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 16 25400.7 70.77
8gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 8 25528.68 68.65
2gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 2 24860.47 68.14
1gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 1 25719.31 67.84
4gb-il Opteron 2374 HE 2.20 [1 processor, 4 cores] 4 25248.93 66.34




Storm on Demand [MI, US]
ID CPU Memory Redis MIOP
48gb Xeon X5650 2.67 [1 processor, 12 cores] 45.9 62269.22 136.4
i5-750-4gb-2disks Core i5 750 2.67 [1 processor, 4 cores] 4 79473.07 124.49
i5-750-2gb-1disk Core i5 750 2.67 [1 processor, 4 cores] 2 81301.01 122.88
x3440-8gb-2disks Xeon X3440 2.53 [1 processor, 8 cores] 8 74553.53 121.15
8gb Xeon X3440 2.53 [1 processor, 4 cores] 7 74183.16 119.81
4gb Core i5 750 2.67 [1 processor, 2 cores] 3.5 76701.91 116.99
e5506x2-4gb-2disks Xeon E5506 2.13 [1 processor, 8 cores] 8 49970.66 83.18
e5506x2-4gb-1disk Xeon E5506 2.13 [1 processor, 8 cores] 8 49760.06 82.8
e5506x2-8gb-4disks Xeon E5506 2.13 [1 processor, 8 cores] 8 49775.14 76.5
32gb Opteron 2378 2.40 [4 processors, 8 cores] 30.4 37672.21 75.82
e5506x2-8gb-4disks Xeon E5506 2.13 [1 processor, 8 cores] 8 49178.7 75.36
2gb Core 2 Quad Q9400 2.66 [1 processor, 1 core] 1.7 22571.9 65
amd2350x2-32gb Opteron 2350 2.00 [4 processors, 8 cores] 32 29966.25 59.15
16gb Opteron 2350 2.00 [2 processors, 4 cores] 15.2 29179.3 52.32




GoGrid [CA, US]
ID CPU Memory Redis MIOP
4gb Xeon E5520 2.27 [1 processor, 4 cores] 4 46176.73 96.12
8gb Xeon E5450 2.99 [1 processor, 6 cores] 8 32590.17 89.74
2gb Xeon E5520 2.27 [1 processor, 2 cores] 2 39756.91 83.74
1gb Xeon E5520 2.27 [1 processor, 1 core] 1 23346.48 79.7




Voxel [New York]
ID CPU Memory Redis MIOP
14gb-ny Xeon L5520 2.26 [7 processors, 7 cores] 14 30566.6 100.02
8gb-ny Xeon L5520 2.26 [4 processors, 4 cores] 8 29376.08 89.78
4gb-ny Xeon L5520 2.26 [2 processors, 2 cores] 4 23520.43 88.62
2gb-ny Xeon L5520 2.26 [1 processor, 1 core] 2 14434.5 79.8




Voxel [Amsterdam]
ID CPU Memory Redis MIOP
14gb-nl Xeon L5520 2.26 [7 processors, 7 cores] 14 27423.78 105.25
8gb-nl Xeon L5520 2.26 [4 processors, 4 cores] 8 30788.54 101.71
4gb-nl Xeon L5520 2.26 [2 processors, 2 cores] 4 27979.96 96.63
2gb-nl Xeon L5520 2.26 [1 processor, 1 core] 2 14691.72 86




Voxel [Singapore]
ID CPU Memory Redis MIOP
14gb-sg Xeon L5520 2.26 [7 processors, 7 cores] 14 28547.88 87.16
8gb-sg Xeon L5520 2.26 [4 processors, 4 cores] 8 27832.58 84.95
4gb-sg Xeon L5520 2.26 [2 processors, 2 cores] 4 24189.04 77.39
2gb-sg Xeon L5520 2.26 [1 processor, 1 core] 2 15237.89 75.99




NewServers [FL, US]
ID CPU Memory Redis MIOP
jumbo Xeon E5504 2.00 [2 processors, 8 cores] 48 39567.78 100
large Xeon E5405 2.00 [1 processor, 4 cores] 4 34997.3 72.25
med Xeon 3.20 [1 processor, 2 cores] 2 22225.77 43.78
small Xeon 2.80 [1 processor, 1 core] 1 11758.45 31.8




Linode VPS Hosting [Atlanta]
ID CPU Memory Redis MIOP
14400-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 14.06 29486.24 99.9
5760-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 5.54 13240.74 96.33
2880-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 2.81 20404.32 93.26
1080-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 1.05 22810.13 87.48
8640-atlanta Xeon L5520 2.27 [2 processors, 4 cores] 8.45 12350.71 86.39




SoftLayer [Dallas]
ID CPU Memory Redis MIOP
4gb-dallas Xeon E5520 2.27 [4 processors, 4 cores] 4 26091.52 98.27
8gb-dallas Xeon E5520 2.27 [8 processors, 8 cores] 8 25067.69 96.93
2gb-dallas Xeon X3460 2.80 [2 processors, 2 cores] 2 22067.26 95.74
1gb-dallas Xeon X3460 2.80 [1 processor, 1 core] 1 14838.57 94.58



SoftLayer [Seattle]
ID CPU Memory Redis MIOP
8gb-seattle Xeon E5520 2.27 [8 processors, 8 cores] 8 25503.99 105.11
4gb-seattle Xeon X3460 2.80 [4 processors, 4 cores] 4 37422.67 103.86
2gb-seattle Xeon X3460 2.80 [2 processors, 2 cores] 2 34444.9 99.62
1gb-seattle Xeon X3460 2.80 [1 processor, 1 core] 1 14836.2 93.01




Terremark vCloud Express [FL, US]
ID CPU Memory Redis MIOP
1gb Opteron 8389 2.91 [1 processor, 1 core] 1 37476.72 66.78
2gb Opteron 8389 2.91 [1 processor, 1 core] 2 37367.9 66.06
16gb-8vpu Opteron 8389 2.91 [1 processor, 8 cores] 16 36601.03 64.74
4gb-2vpu Opteron 8389 2.91 [1 processor, 2 cores] 4 42788.76 64.03
8gb-4vpu Opteron 8389 2.91 [1 processor, 4 cores] 8 41525.25 61.08




VPS.NET [Atlanta]
ID CPU Memory Redis MIOP
12node-atl Xeon E5520 2.27 [5 processors, 5 cores] 4.4 28676.91 98.17
18node-atl Xeon E5520 2.27 [7 processors, 7 cores] 6.6 17864.39 88.83
3node-atl Xeon E5520 2.27 [2 processors, 2 cores] 1.1 25017.57 81
6node-atl Xeon E5520 2.27 [3 processors, 3 cores] 2.2 23600.54 73.24




OpSource Cloud [VA, US]
ID CPU Memory Redis MIOP
1gb Xeon X7460 2.66 [1 processor, 1 core] 1 13552.35 38.13
2gb Xeon X7460 2.66 [1 processor, 1 core] 2 12775.31 36.56
4gb-2cpu Xeon X7460 2.66 [1 processor, 2 cores] 4 13772.58 36.52
8gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 8 13538.06 34.62
16gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 16 13131.61 33.88
32gb-4cpu Xeon X7460 2.66 [1 processor, 4 cores] 32 12260.02 32.99




Speedyrails [Quebec]
ID CPU Memory Redis MIOP
1gb Xeon E5520 2.27 [2 processors, 8 cores] 1 20705.27 92.89
4gb Xeon E5520 2.27 [2 processors, 8 cores] 4 21118.42 87.52
2gb Xeon E5520 2.27 [2 processors, 8 cores] 2 19086.71 83.5




Zerigo [CO, US]
ID CPU Memory Redis MIOP
4gb Opteron 2374 HE 2.20 [2 processors, 4 cores] 4 16428.46 63.81




ReliaCloud [MN, US]
ID CPU Memory Redis MIOP
4gb Xeon E5504 2.00 [1 processor, 4 cores] 4 6499.32 34.04
2gb Xeon E5504 4.40 [1 processor, 2 cores] 2 5559.85 32.21
1gb Xeon E5504 2.00 [1 processor, 1 core] 1 5300.22 22.42




Gandi.net [France]
ID CPU Memory Redis MIOP
4gb Opteron 8346 HE 1.79 [1 processor, 4 cores] 4 33143.29 46.59
6gb Opteron 8346 HE 1.79 [1 processor, 6 cores] 6 10647.31 43.35
2gb Opteron 8346 HE 1.79 [1 processor, 2 cores] 2 10713.03 36.56
1gb Opteron 8346 HE 1.79 [1 processor, 1 core] 1 11255.61 33.6




CloudSigma [Switzerland]
ID CPU Memory Redis MIOP
16gb-16gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 16 37227.13 72.03
32gb-20gh Six-Core Opteron 2427 2.21 [1 processor, 8 cores] 32 36628.6 67.26
4gb-4gh Six-Core Opteron 2427 2.21 [1 processor, 2 cores] 4 37030.85 60.68
2gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 2 21457.67 56.7
1gb-2gh Six-Core Opteron 2427 2.21 [1 processor, 1 core] 1 21682.06 55.96




IBM Development & Test Cloud [NY, US]
ID CPU Memory Redis MIOP
dev-large Xeon X5570 2.93 [1 processor, 8 cores] 3.5 62280.44 124.37
dev-med Xeon X5570 2.93 [1 processor, 4 cores] 1.75 46424.68 95.5




BlueLock [IN, US]
ID CPU Memory Redis MIOP
16gb-8cpu Xeon X5550 2.67 [1 processor, 8 cores] 16 57578.66 117.88
8gb-4cpu Xeon X5550 2.67 [1 processor, 4 cores] 8 48148.66 115.64
4gb-2cpu Xeon X5550 2.67 [1 processor, 2 cores] 4 57390.96 107.8
2gb Xeon X5550 2.67 [1 processor, 1 core] 2 29641.05 103.73
1gb Xeon X5550 2.67 [1 processor, 1 core] 1 24710.52 91.9




Cloud Central [AU]
ID CPU Memory Redis MIOP
large Opteron 2374 HE 2.20 [4 processors, 4 cores] 8 18585.76 60.91
reg Opteron 2374 HE 2.20 [4 processors, 4 cores] 4 19328.25 59.37
huge Opteron 2374 HE 2.20 [4 processors, 4 cores] 16 13726.38 57.93
med Opteron 2374 HE 2.20 [4 processors, 4 cores] 2 18325.34 57.23
small Opteron 2374 HE 2.20 [4 processors, 4 cores] 1 17260.76 56.05




RimuHosting [TX, US]
ID CPU Memory Redis MIOP
tx-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 24771.2 71.62




RimuHosting [NZ]
ID CPU Memory Redis MIOP
nz-2gb Xeon E5506 2.13 [1 processor, 2 cores] 2 22446.26 71.22




ElasticHosts [UK]
ID CPU Memory Redis MIOP
4gb-8gh Xeon E5420 2.50 [1 processor, 4 cores] 4 38376.45 74.91
1gb-2gh Xeon E5420 2.50 [1 processor, 1 core] 1 17456.46 67.03
2gb-4gh Xeon E5420 2.50 [1 processor, 2 cores] 2 24852.55 66.4




Flexiscale [UK]
ID CPU Memory Redis MIOP
8gb Opteron 8218 2.62 [1 processor, 4 cores] 8 20853.61 44.9
4gb Dual-Core Opteron 8220 2.80 [1 processor, 2 cores] 4 22004.85 44.51
2gb Opteron 8218 2.60 [1 processor, 1 core] 2 17149.07 42.86
1gb Opteron 8218 2.60 [1 processor, 1 core] 1 16410.28 37.14

06/12/2010 12:33 PDT

Cloud Server Benchmarking Part 3: Java, Ruby, Python & PHP

This is the third post in our series on cloud server performance benchmarking. The previous blog posts, What is an ECU? CPU Benchmarking in the Cloud and Disk IO Benchmarking in the Cloud focused on more synthetic (i.e. raw performance numbers with no real-world application) CPU and Disk IO performance. In this post, we'll look at performance using 4 common interpreted (full or byte-code) programming languages: Java, Ruby, Python and PHP.

This post is by no means intended to provide exhaustive and exact metrics for cloud server performance with these languages. The purpose of this and the other benchmarking posts is to provide a quick reference and starting point for further research and evaluation of different cloud providers. There is much variation in performance, features, reliability, support, pricing and other factors of different cloud providers, yet if you were to evaluate those providers based solely on marketing literature, you'd be hard pressed to distinguish one from another. Our goal is to reduce this ambiguity by providing objective, quantifiable measurements for comparing cloud providers.

Benchmark Setup
All benchmarked cloud servers were configured almost identically in terms of OS and software:
  • Operating System: CentOS 64-bit (except for IBM Developer Cloud using RHEL 5 32-bit)
  • File System: ext3
  • Sun JDK 1.6
  • Ruby 1.8.5
  • Python 2.4.3
  • PHP 5.1.6
Benchmark Methodology
We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is Storm on Demand's bare-metal instance on E5506 2.13 hardware (dual processors - 8 cores total), with 4 x 15K RPM SAS drives (Raid 10) and 8GB ram. This is a fairly high-end server with many cores and very fast IO. We assigned this server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 4 language benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Example Score Calculation:
Baseline aggregate score = SPECjvm2008: 100; ruby-benchmark-suite: 100; PyBench: 100; PHPBench 100
Total baseline score = 400

Server X aggregate score = SPECjvm2008: 80/100; ruby-benchmark-suite: 90/100; PyBench: 70/100; PHPBench 95/100
Total baseline score = 335
Server X Score = (335/400) * 100 = 83.75

Benchmarks
The following benchmarks were used in this post:
  • SPECjvm (Java Virtual Machine Benchmark) [Higher score is better]: A benchmark suite for measuring the performance of a Java Runtime Environment (JRE), containing several real life applications and benchmarks focusing on core java functionality. The suite focuses on the performance of the JRE executing a single application; it reflects the performance of the hardware processor and memory subsystem, but has low dependence on file I/O and includes no network I/O across machines. The SPECjvm2008 workload mimics a variety of common general purpose application computations. These characteristics reflect the intent that this benchmark will be applicable to measuring basic Java performance on a wide variety of both client and server systems.
  • Ruby [Lower score is better]: A suite for measuring the performance of Ruby implementations, including micro-benchmarks that focus on core Ruby functionality, as well as macro-benchmarks that represent a variety of real, common workloads. The project is aimed at providing a useful suite for comparing the performance of the available Ruby implementations and, as a positive side effect, to give VM implementers an additional tool to measure and identity performance related issues. The score for this benchmark is the average of the median times to execute 13 of the macro benchmarks (all macro benchmarks minus bm_hilbert_matrix.rb). Specific times for these benchmarks individually are available in the benchmark source file.
  • Python [Lower score is better]: A collection of tests that provides a standardized way to measure the performance of Python implementations. The score for this benchmark is the average of all benchmarks performed by pybench.
  • PHP [Higher score is better]: A benchmark suite for PHP. It performs a large number of simple tests in order to benchmark various aspects of the PHP interpreter. PHPBench can be used to compare hardware, operating systems, PHP versions, PHP accelerators and caches, compiler options, etc.
Results
The results are divided into tables separated by provider. If the provider has more than one data center locations, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), PHPBench score, PyBench score, ruby-benchmark-suite score, SPECjvm2008 score, and the aggregate baseline relative score (as described above). Each of the 4 individual scores are linked to the complete results for those specific tests (a single small text file for PHPBench, PyBench, ruby-benchmark-suite and a 3-4MB zip file for SPECjvm2008).

Note: Some smaller server configurations (e.g. EC2 m1.small and c1.medium) are not included in these results due to their inability to run one more more of the individual benchmarks.


Amazon Web Services (AWS) [US East]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50328 3725 4.12 197.42 138.58
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50253 3737 4.08 108.09 115.45
c1.xlarge Xeon E5410 2.33 8 cores 7 44460 4586 5.46 131.12 105.76
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50396 3756 4.1 58.15 102.27
m1.xlarge Xeon E5430 2.66 4 cores 15 38737 5279 6.45 68.09 79.62
m1.large Xeon E5430 2.66 2 cores 7.5 38625 5324 6.58 38.21 71.29



Amazon Web Services (AWS) [US West]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50553 3723 4.09 197.83 139.02
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50062 3797 4.1 108.53 114.82
c1.xlarge Xeon E5410 2.33 8 cores 7 44733 4566 5.48 130.26 105.75
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50832 3724 4.12 58.59 102.81
m1.xlarge Xeon E5430 2.67 4 cores 15 39001 5323 6.36 67.98 79.79
m1.large Xeon E5430 2.66 2 cores 7.5 38716 5382 6.46 38.17 71.38



Amazon Web Services (AWS) [EU West]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50775 3739 4.1 200.09 139.55
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50207 3723 4.1 107.32 115.22
c1.xlarge Xeon E5410 2.33 8 cores 7 44656 4563 5.51 131.52 105.94
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50275 3732 4.11 58.92 102.55
m1.xlarge Xeon E5430 2.67 4 cores 15 38387 5259 6.37 69.06 79.94
m1.large Xeon E5430 2.67 2 cores 7.5 38327 5299 6.39 39.15 71.91



Amazon Web Services (AWS) [APAC]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50336 3743 4.42 131.72 119.05
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50095 3762 4.42 89.86 108.51
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50042 3799 4.43 50.63 97.92
c1.xlarge Xeon E5410 2.33 8 cores 7 44992 4608 5.57 63.93 88.13
m1.xlarge Xeon E5430 2.67 4 cores 15 38793 5314 6.56 54.15 75.62
m1.large Xeon E5430 2.67 2 cores 7.5 38214 5299 6.49 34.1 70.28



Rackspace Cloud [Dallas]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 2374 HE 2.20 4 cores 4 39210 5465 5.74 63.07 79.76
16gb Opteron 2374 HE 2.20 4 cores 16 37914 5517 5.75 62.77 78.66
1gb Opteron 2374 HE 2.20 4 cores 1 38679 5396 5.81 56.22 77.73
2gb Opteron 2374 HE 2.20 4 cores 2 38868 5454 5.87 41.89 73.71



Rackspace Cloud [Chicago]
ID CPU Memory PHP Python Ruby SPECjvm Score
16gb-il Opteron 2374 HE 2.20 4 cores 16 38729 5397 5.7 63.86 80.08
4gb-il Opteron 2374 HE 2.20 4 cores 4 39065 5424 5.8 62.14 79.42
8gb-il Opteron 2374 HE 2.20 4 cores 8 38958 5455 5.66 63.88 78.31
1gb-il Opteron 2374 HE 2.20 4 cores 1 38443 5434 5.68 56.88 77.99



Storm on Demand [MI, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
48gb Xeon X5650 2.67 12 cores 45.9 50602 3810 6.17 174.81 124.05
i5-750-4gb-2disks Intel i5 750 2.67 4 cores 4 45560 4326 3.17 89.03 110.35
i5-750-2gb-1disk Intel i5 750 2.67 4 cores 2 45620 4352 3.16 82.16 108.53
x3440-8gb-2disks Xeon X3440 2.53 8 cores 8 43428 4599 3.38 97.18 107.55
8gb Xeon X3440 2.53 4 cores 7 43861 4565 3.35 86.55 105.51
e5506x2-4gb-1disk Xeon E5506 2.13 8 cores 8 40577 4774 3.93 108.06 103.52
e5506x2-4gb-2disks Xeon E5506 2.13 8 cores 8 40136 4766 3.94 105.9 102.67
32gb Opteron 2378 2.40 8 cores 30.4 40954 5104 3.8 104.01 101.96
4gb Core i5 750 2.67 2 cores 3.5 45983 4331 3.17 49.71 100.35
e5506x2-8gb-4disks Xeon E5506 2.13 8 cores 8 40731 4765 4.02 96.18 100
e5506x2-8gb-4disks Xeon E5506 2.13 8 cores 8 40306 4780 4.01 95.13 99.45
2gb Core 2 Quad Q9400 2.66 1 core 1.7 49905 4142 3.59 23.91 93.6
amd2350x2-32gb-4disks Opteron 2350 2.00 8 cores 32 32182 6271 4.96 86.73 81.55
16gb Opteron 2350 2.00 4 cores 15.2 32240 6187 4.87 53.1 73.48



GoGrid [CA, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5450 2.99 6 cores 8 55970 3616 3.53 101.76 121.46
4gb Xeon E5520 2.27 4 cores 4 40513 4806 3.8 74.34 95.42
2gb Xeon E5520 2.27 2 cores 2 38206 5090 4.09 36.13 80.82
1gb Xeon E5520 2.27 1 core 1 38215 4945 4.02 20.56 77.89



Voxel [New York]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-ny Xeon L5520 2.26 7 cores 14 44463 4214 4.98 119.82 106.89
8gb-ny Xeon L5520 2.26 4 cores 8 44043 4237 4.91 81.23 96.73
4gb-ny Xeon L5520 2.26 2 cores 4 40921 4446 4.95 45.71 84.1
2gb-ny Xeon L5520 2.26 1 core 2 44022 4210 4.94 24.47 82.02



Voxel [Amsterdam]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-nl Xeon L5520 2.26 7 cores 14 45069 4221 4.85 124.22 108.9
8gb-nl Xeon L5520 2.26 4 cores 8 45043 4219 4.9 82.02 97.71
4gb-nl Xeon L5520 2.26 2 cores 4 44880 4217 4.83 45.52 88.44
2gb-nl Xeon L5520 2.26 1 core 2 44436 4229 4.86 24.57 82.51



Voxel [Singapore]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-sg Xeon L5520 2.26 7 cores 14 42488 4468 5.14 111.08 101.17
8gb-sg Xeon L5520 2.26 4 cores 8 42355 4473 5.15 75.75 91.84
4gb-sg Xeon L5520 2.26 2 cores 4 42433 4480 5.15 42.15 83.11
2gb-sg Xeon L5520 2.26 1 core 2 42153 4482 5.16 23.08 77.93



NewServers [FL, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
fast Xeon E5450 2.99 4 cores 4 57795 3600 3.24 81.3 120.71
jumbo Xeon E5504 2.00 8 cores 48 37593 5087 4.2 110.03 99.02
large Xeon E5405 2.00 4 cores 4 38161 5462 4.59 59.75 82.66
med Xeon 3.20 2 cores 2 29381 7383 5.89 25.74 57.92
small Xeon 2.80 1 core 1 25223 8183 6.9 12.37 47.82



Linode VPS Hosting [Atlanta]
ID CPU Memory PHP Python Ruby SPECjvm Score
5760 Xeon L5520 2.27 4 cores 5.54 42841 4418 4.67 78.51 95.19
14400 Xeon L5520 2.27 4 cores 14.06 42644 4410 4.65 78.45 95.19
2880 Xeon L5520 2.27 4 cores 2.81 39050 4503 5.02 66.78 87.8
1080 Xeon L5520 2.27 4 cores 1.05 40442 4579 5.12 59.57 85.95
8640 Xeon L5520 2.27 4 cores 8.45 32904 5632 5.28 67.85 78.02



SoftLayer [Dallas]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42414 4439 5.5 123.82 103.33
4gb Xeon E5520 2.27 4 cores 4 42471 4459 5.51 78.74 91.49
2gb Xeon X3460 2.80 2 cores 2 47816 4130 5.53 49.59 89.26
1gb Xeon X3460 2.80 1 core 1 47634 4118 4.61 27.86 87.21



SoftLayer [WDC]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42981 4441 5.48 127.55 104.7
4gb Xeon X3460 2.80 4 cores 4 47425 4101 4.62 91.07 103.58
2gb Xeon X3460 2.80 2 cores 2 47478 4108 4.4 48.95 93.7
1gb Xeon X3460 2.80 1 core 1 47614 4113 4.47 27.61 87.85



SoftLayer [Seattle]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42642 4451 5.54 125.75 103.76
2gb Xeon X3460 2.80 2 cores 2 48095 4138 4.63 48.7 92.67
1gb Xeon X3460 2.80 1 core 1 47472 4113 4.61 26.49 86.79



Terremark vCloud Express [FL, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
16gb-8vpu Opteron 8389 2.91 8 cores 16 48625 4623 3.99 100.42 106.9
1gb Opteron 8389 2.91 1 core 1 48997 4254 3.18 22 95.4
2gb Opteron 8389 2.91 1 core 2 48651 4321 3.17 21.87 94.82
4gb-2vpu Opteron 8389 2.91 2 cores 4 39078 5019 3.72 38.78 84.82
8gb-4vpu Opteron 8389 2.91 4 cores 8 32959 5388 4.19 70.56 84.67



VPS.NET [Atlanta]
ID CPU Memory PHP Python Ruby SPECjvm Score
18node Xeon E5520 2.27 7 cores 6.6 39447 4762 5.85 96.35 91.45
12node Xeon E5520 2.27 [5 processors, 5 cores 4.4 42843 5035 5.39 77.12 88.65
6node Xeon E5520 2.27 [3 processors, 3 cores 2.2 33213 5730 6.8 45.81 67.86
3node Xeon E5520 2.27 2 cores 1.1 34188 5452 6.54 35.72 67.49



OpSource Cloud [VA, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
2gb Xeon X7460 2.66 1 core 2 28083 7721 7.93 11.13 48.23
8gb-4cpu Xeon X7460 2.66 4 cores 8 24639 8758 9.08 9.03 42.14
32gb-4cpu Xeon X7460 2.66 4 cores 32 23604 8790 8.87 9.04 41.72
16gb-4cpu Xeon X7460 2.66 4 cores 16 24636 8738 9.43 8.77 41.69



Speedyrails [Quebec]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Xeon E5520 2.27 8 cores 4 41819 4735 5.26 100.74 96.12
1gb Xeon E5520 2.27 8 cores 1 41541 4975 5.28 56.54 83.18



Zerigo [CO, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 2374 HE 2.20 4 cores 4 39030 5400 5.58 55.54 78.46



ReliaCloud [MN, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Xeon E5504 2.00 4 cores 4 13287 13209 10.7 3.85 27.56
8gb 8 14159 15567 10.79 3.69 26.62
2gb Xeon E5504 4.40 2 cores 2 12131 13819 11.36 5.36 26.31



CloudSigma [Switzerland]
ID CPU Memory PHP Python Ruby SPECjvm Score
32gb-20gh Six-Core Opteron 2427 2.21 8 cores 32 37912 5646 4.27 97.2 93.17
16gb-16gh Six-Core Opteron 2427 2.21 8 cores 16 37599 5654 4.3 98.23 93.05
8gb-8gh Six-Core Opteron 2427 2.21 4 cores 8 37473 5619 4.2 59.99 83.72
4gb-4gh Six-Core Opteron 2427 2.21 2 cores 4 37409 5582 4.26 32.54 76.35
1gb-2gh Six-Core Opteron 2427 2.21 1 core 1 37326 5609 4.27 17.2 72.16
2gb-2gh Six-Core Opteron 2427 2.21 1 core 2 36561 5789 4.25 17.82 71.3



IBM Development & Test Cloud [NY, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
large Xeon X5570 2.93 8 cores 3.5 45260 4711 3.6 140.6 117.53



BlueLock [IN, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb-4cpu Xeon X5550 2.67 4 cores 8 54399 3761 3 94.08 123.02
16gb-8cpu Xeon X5550 2.67 8 cores 16 48314 3859 3.36 96.07 115.41
4gb-2cpu Xeon X5550 2.67 2 cores 4 45071 3894 3.21 51.9 103.05
2gb Xeon X5550 2.67 1 core 2 50021 3739 3.16 26.72 101.31
1gb Xeon X5550 2.67 1 core 1 49499 3745 3.06 16.01 99.2



Cloud Central [Australia]
ID CPU Memory PHP Python Ruby SPECjvm Score
reg Opteron 2374 HE 2.20 4 cores 4 38725 5469 5.55 60.59 79.41
large Opteron 2374 HE 2.20 4 cores 8 38921 5439 5.64 60.58 79.36
huge Opteron 2374 HE 2.20 4 cores 16 37989 5573 6.43 61.86 76.4
small Opteron 2374 HE 2.20 4 cores 1 37356 5597 6.18 49.6 73.36



RimuHosting [TX, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
tx-2gb Xeon E5506 2.13 2 cores 2 39096 4883 4.89 39.28 79.16



RimuHosting [New Zealand]
ID CPU Memory PHP Python Ruby SPECjvm Score
nz-2gb Xeon E5506 2.13 2 cores 2 39529 4854 5.08 39.22 78.78



ElasticHosts [UK]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb-20gh Xeon E5420 2.50 6 cores 8 47168 4431 4.1 92.82 104.47
2gb-4gh Xeon E5420 2.50 2 cores 2 43723 4508 4.11 34.92 86.79
1gb-2gh Xeon E5420 2.50 1 core 1 44943 4470 4.1 20.94 84.19



Flexiscale [UK]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 8220 2.80 2 cores 4 36319 6033 4.03 34.1 69.57
8gb Opteron 8218 2.62 4 cores 8 33878 6570 4.35 53.46 67.87
2gb Opteron 8218 2.60 1 core 2 35141 6505 4.16 17.19 63.09
1gb Opteron 8218 2.60 1 core 1 34497 6590 4.39 16.16 63.05




Summary
We think it is important for these and other cloud performance benchmarks to be made available publicly. With all the hype surrounding cloud computing today, new cloud users may not understand what exactly they are getting in terms of performance and scalability. As these benchmarks show, not all clouds are created equal, and there can be very pronounced performance variations from one provider to another. One of the greatest benefits of the cloud is simplicity and a perceived ability to infinitely scale. However, it should be understood that with that simplicity and abstraction, the cloud is still powered by physical hardware, CPUs, disks and memory, and subject to the same performance limitations that same hardware exhibits in a non-cloud environment.

06/11/2010 15:28 PDT

Google Storage, a CDN/Storage Hybrid?

We received our Google Storage for Developers account today, and being the curious types, immediately ran some network tests. We maintain a network of about 40 cloud servers around the world which we use to monitor cloud servers, test cloud-to-cloud network throughput, and our cloud speedtest. We used a handful of these to test Google Storage uplink and downlink throughput and latency. We were very surprised by the low latency and consistently fast downlink throughput to most locations.

Most storage services like Amazon's S3 and Azure Blob Storage are physically hosted from a single geographical location. S3 for example, is divided into 4 regions, US West, US East, EU West and APAC. If you store a file to an S3 US West bucket, it is not replicated to the other regions, and can only be downloaded from that regions' servers. The result is much slower network performance from locations with poor connectivity to that region. Hence, you need to chose your S3 region wisely based on geographical and/or network proximity. Azure's Blob storage uses a similar approach. Users are able to add CDN on top of those services (CloudFront and Azure CDN), but the CDN does not provide the same access control and consistency features of the storage service.

In contrast, Google's new Storage for Developers service appears to store uploaded files to a globally distributed network of servers. When a file is requested, Google uses some DNS magic to direct a user to a server that will provide the fastest access to that file. This is very similar to the approach that Content Delivery Networks like Akamai and Edgecast work, wherein files are distributed to multiple globally placed PoPs (point-of-presence).

Our simple test consisted of requesting a test 10MB file from 17 different servers located in the US, EU and APAC. The test file was set to public and we used wget to test downlink and Google's gsutil to test uplink throughput (wget was faster than gsutil for downloads). In doing so, we found the same test URL resolved to 11 different Google servers with an average downlink of about 40 Mb/s! This hybrid model of CDN-like performance with enterprise storage features like durability, consistency and access control represents an exciting leap forward for cloud storage!

Google Storage Network Performance Tests
Service Location Resolved To Latency Upload (Mb/s) Download (Mb/s)
Gigenet IL, US 209.85.225.132 15.05 15.13 68.4
CloudSigma Switzerland 209.85.227.132 24.25 1.75 21.5
Linode (London) UK 209.85.229.132 7.13 5.72 41.6
Bluelock IN, US 64.233.169.132 23.1 4.42 56.8
EC2 (EU West) Ireland 66.102.9.132 2.19 6.58 29.5
CloudCentral Australia 66.102.11.132 5.96 4.84 8.2
Rimu Hosting New Zealand 66.102.11.132 27.05 5.53 9.64
Gandi Cloud VPS France 66.102.13.132 19.9 2.81 31.1
Zerigo CO, US 72.14.203.132 57.05 5.63 24.04
Voxel (Amsterdam) The Netherlands 72.14.204.132 83.2 3.86 40.6
EC2 (US East) VA, US 72.14.204.132 2.72 10.14 39.4
EC2 (US West) VA, US 72.14.204.132 27.7 8.77 9.5
Speedyrails Canada 72.14.204.132 24.5 8.28 33.1
VPS.net (UK) UK 74.125.79.132 11.6 3.48 32.1
Rackspace (Chicago) IL, US 74.125.95.132 13.05 4.23 70.3
GoGrid CA, US 74.125.127.132 24.2 6.32 39.9
Zerigo CO, US 74.125.155.132 55.3 10.51 42.08

06/05/2010 19:22 PDT

Disk IO Benchmarking in the Cloud

This is the second post in our series on cloud server performance benchmarking. The previous blog post, What is an ECU? CPU Benchmarking in the Cloud, focused strictly on CPU performance. In this post, we'll look at disk IO performance. Choosing a cloud provider should be based on a many factors including performance, price, support, reliability/uptime, scalability, network performance and features. Our intent is to provide a good reference for those looking to use cloud services by providing objective analysis in regards to all of these factors.

Benchmark Setup
All benchmarked cloud servers were configured almost identically in terms of OS and software, CentOS 5.4 64-bit (or 32-bit in the case of EC2 m1.small and c1.medium and IBM's Development Cloud where 64-bit is not supported). File systems were formatted ext3.

Benchmark Methodology
In the previous post we used Amazon's EC2 ECU (Elastic Compute Unit) as a baseline for comparison of CPU performance between providers. In terms of disk performance, there really isn't a common term synonymous to the ECU. However, most readers will be at least be somewhat familiar with hardware disk IO performance factors such drive types (SAS, SATA), spindle speeds (10K, 15K) and RAID levels. So, we chose to use a "bare-metal" cloud server as the baseline for disk IO performance in this post. Our experience has been that most providers will not disclose much technical detail about their underlying storage systems. However, in the case of Storm on Demand's new Bare Metal Cloud Servers they fully disclose most technical details about their servers (one of the selling points of this service). With this service, you are assigned a dedicated server. Your OS still runs on a hypervisor (Storm uses Xen), but the underlying hardware is not shared with any other virtual servers, and you thereby have the full resources of that server available (no CPU limits, disk or memory sharing).

The server model we chose as the performance comparison baseline is the dual processor Intel E5506 2.13 GHz (8 cores total) with 4 x 15K RPM SAS drives configured in hardware managed Raid 1+0. SAS 15K is one of the fastest storage interfaces with throughput up to 6 Gbp/s, and Raid 1+0 can improve performance using striping.

To use this server as the baseline we assigned it an aggregate IO Performance score (IOP) of exactly 100 points. Other server disk IO benchmark results were then compared to baseline results and assigned a relative score, where 100 is equal in performance, less than 100 worse, and greater than 100 better. For example, a server with a score of 50 scored 50% lower than the baseline server, while a server with a score of 125, scored 25% higher.

To compute the IOP, the results from each of the benchmarks is first calculated for the baseline server. Each benchmark has a weight for the overall aggregate score. The baseline server benchmark scores are the 100% mark for each of these weights (i.e. the baseline server receives the full weight of each benchmark for its IOP). Once the weights are calculated, they are then summed to create an aggregate score. This score is then compared with the aggregate score of the baseline server and use to produce the IOP.

Example IOP Calculation:
Baseline aggregate score = blogbench-read: 200/200; bonnie++: 100/100; dbench: 30/30; fio: 30/30; hdparm: 100/100; iozone: 200/200; tiobench: 30/30; Total aggregate score = 690
Baseline IOP = (690/690) * 100 = 100
Server X aggregate score = blogbench-read: 180/200; bonnie++: 80/100; dbench: 25/30; fio: 20/30; hdparm: 90/100; iozone: 175/200; tiobench: 28/30; Total aggregate score = 598
Server X IOP = (598/690) * 100 = 86.67

Benchmarks
We used a combination 7 disk IO performance benchmarks to create the IOP score. The following is a description of the benchmarks and corresponding weights use:
  • Blogbench [weight=200]: BlogBench is designed to replicate the load of a real-world busy file server by stressing the file-system with multiple threads of random reads, writes, and rewrites. The behavior is mimicked of that of a blog by creating blogs with content and pictures, modifying blog posts, adding comments to these blogs, and then reading the content of the blogs. All of these blogs generated are created locally with fake content and pictures.
  • Bonnie++ [weight=100]: Bonnie++ is based on the Bonnie hard drive benchmark by Tim Bray. This program is used by ReiserFS developers, but can be useful for anyone who wants to know how fast their hard drive or file system is.
  • Dbench (128 clients) [weight=30]: Dbench is a benchmark designed by the Samba project as a free alternative to netbench, but dbench contains only file-system calls for testing the disk performance.
  • Flexible IO Tester (fio) [weight=30]: fio is an I/O tool meant to be used both for benchmark and stress/hardware verification. It has support for 13 different types of I/O engines (sync, mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio, and more), I/O priorities (for newer Linux kernels), rate I/O, forked or threaded jobs, and much more.
  • hdparm buffered disk reads [weight=100]: Determines the speed of reading through the buffer cache to the disk without any prior caching of data. This measurement is an indication of how fast the drive can sustain sequential data reads under Linux, without any filesystem overhead.
  • IOzone (4GB reads & writes) [weight=200]: The IOzone benchmark tests the hard disk drive/file-system performance.
  • Threaded I/O Tester (64 MB random write; write; read) [weight=30]: Tiotester (Threaded I/O Tester) benchmarks the hard disk drive / file-system performance.
We credit the Phoronix Test Suite for making it easier to run the benchmarks. The tests above come from the Disk test suite (except for bonnie++). If you'd like to compare your own server to the baseline server in this post, you can install Phoronix and use the comparison feature when running your own IO benchmarks. The full baseline server results are available here (including the ID for comparison tests). The baseline server bonnie++ results are available here.

Results
The following results are divided into sections based on provider. If the provider has more than one data center, multiple tables are displayed one for each. Each table shows the server identifier, CPU architecture, memory (GB), storage description (if known) or size, the server price, and the IOP score.

Amazon EC2 servers can use either ephemeral (local host) or external Elastic Block Storage (EBS) storage. Regarding EBS, Amazon states "The latency and throughput of Amazon EBS volumes is designed to be significantly better than the Amazon EC2 instance stores in nearly all cases.". EBS is a block level off-instance storage, meaning it exists independent of the underlying host. Striping of multiple EBS volumes can be used to improve performance. We used a single EBS volume only (no striping or other performance enhancements).

Because EBS is off-instance, it has the advantage of added durability because if the host system fails the instance can be quickly restarted on another host (this is not automatic). EBS volumes are automatically replicated to prevent data loss due to failure of any single hardware component. EBS is billed at $0.10/GB/month + $0.10 per 1 million I/O requests. The prices below do not include EBS usage charges.

Amazon states that larger instances will see faster and more consistent throughput performance. We found this to be generally true, until we got into the m2 instances where larger instance performance increase was not apparent.

Following suit with the CPU benchmarks, we saw a notable difference in performance in the APAC region compared with instances of the same size in other regions.

We've noted the standard EC2 hourly pricing in the tables. EC2 also offers reserve pricing (pay a 1 or 3 year setup in exchange for significantly discounted hourly rates) and spot/bid pricing.

Amazon Web Services (AWS) [US East]
ID CPU Memory Storage Price IOP
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 2.4/hr 96.88
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.5/hr 88.9
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.4/hr 87.56
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.68/hr 70.88
m1.xlarge Xeon E5430 2.66 4 cores 15 EBS 0.68/hr 57.03
m1.large Xeon E5430 2.66 2 cores 7.5 EBS 0.34/hr 53.01
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.17/hr 34.82
m1.small Opteron 2218 2.60 1 core 1.7 EBS 0.085/hr 22.2


Amazon Web Services (AWS) [US West]
ID CPU Memory Storage Price IOP
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.57/hr 104.36
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 1.34/hr 97.87
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.68/hr 89.56
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.76/hr 59.53
m1.xlarge Xeon E5430 2.67 4 cores 15 EBS 0.76/hr 58.45
m1.large Xeon E5430 2.66 2 cores 7.5 EBS 0.38/hr 49.11
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.19/hr 38.65
m1.small Xeon E5430 2.66 1 core 1.7 EBS 0.095/hr 26.47


Amazon Web Services (AWS) [EU West]
ID CPU Memory Storage Price IOP
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 1.34/hr 92.66
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.57/hr 91.82
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.68/hr 81.91
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.76/hr 71.31
m1.large Xeon E5430 2.67 2 cores 7.5 EBS 0.38/hr 54.81
m1.xlarge Xeon E5430 2.67 4 cores 15 EBS 0.76/hr 54.48
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.19/hr 35.98
m1.small Xeon E5430 2.66 1 core 1.7 EBS 0.095/hr 22.28


Amazon Web Services (AWS) [APAC]
ID CPU Memory Storage Price IOP
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 EBS 1.34/hr 81.59
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 EBS 2.68/hr 80.81
m2.xlarge Xeon X5550 2.67 2 cores 17.1 EBS 0.57/hr 76.44
c1.xlarge Xeon E5410 2.33 8 cores 7 EBS 0.76/hr 68.68
m1.large Xeon E5430 2.67 2 cores 7.5 EBS 0.38/hr 66.12
m1.xlarge Xeon E5430 2.67 4 cores 15 EBS 0.76/hr 57.47
c1.medium Xeon E5410 2.33 2 cores 1.7 EBS 0.19/hr 35.43
m1.small Xeon E5430 2.67 1 core 1.7 EBS 0.095/hr 27.89



Rackspace Cloud
Rackspace Cloud uses local storage for cloud servers. This provides generally good performance on nodes of all sizes. However, the tradeoff is that it does not provide durability should the host system fail. Rackspace does offers scheduled hot backup/imaging capabilities for added durability.

Rackspace Cloud [Dallas]
ID CPU Memory Storage Price IOP
rs-16gb Opteron 2374 2.20 4 cores 16 620 GB 0.96/hr 86.75
rs-4gb Opteron 2374 2.20 4 cores 4 160 GB 0.24/hr 63.1
rs-2gb Opteron 2374 2.20 4 cores 2 80 GB 0.12/hr 57.91
rs-1gb Opteron 2374 2.20 4 cores 1 40 GB 0.06/hr 57.36


Rackspace Cloud [Chicago]
ID CPU Memory Storage Price IOP
rs-8gb-il Opteron 2374 2.20 4 cores 8 320 GB 0.48/hr 77.95
rs-16gb-il Opteron 2374 2.20 4 cores 16 620 GB 0.96/hr 74.43
rs-4gb-il Opteron 2374 2.20 4 cores 4 160 GB 0.24/hr 72.12
rs-2gb-il Opteron 2374 2.20 4 cores 2 80 GB 0.12/hr 56.33
rs-1gb-il Opteron 2374 2.20 4 cores 1 40 GB 0.06/hr 45.84


Storm on Demand
Storm offers a diverse range of cloud servers to chose from including traditional cloud servers and their new "bare metal" dedicated cloud servers. The row highlighted in green is the baseline server for the IOP metric in this post. The 48GB cloud server was the top performer at almost 70% faster than the baseline server! We aren't sure how they do this, but we do know that the Intel
Westmere CPU it runs on is very high performing (it was also the top performer in the CPU benchmarks post), and most likely the storage components are very high end as well. To our knowledge, all Storm's cloud servers run off of local host storage. They offer automated daily backups and imaging for added durability.

Storm Cloud [MI, US]
ID CPU Memory Storage Price IOP
storm-48gb Xeon X5650 2.67 12 cores 45.9 1500 GB 1.37/hr 168.48
e5506x2-8gb Xeon E5506 2.13 8 cores 8 4 SAS Raid 10 261GB 0.48/hr 100
storm-8gb Xeon X3440 2.53 4 cores 7 600 GB 0.27/hr 96.94
storm-32gb Opteron 2378 2.40 8 cores 30.4 1500 GB 0.69/hr 90.99
e5506x2-8gb Xeon E5506 2.13 8 cores 8 4 SATA Raid 10 917GB 0.391/hr 90.3
x3440-8gb Xeon X3440 2.53 8 cores 8 2 SATA Raid 1 919GB 0.274/hr 78.62
amd2350x2-32gb Opteron 2350 2.00 8 cores 32 4 SATA Raid 10 1853GB 0.713/hr 77.61
storm-16gb Opteron 2350 2.00 4 cores 15.2 750 GB 0.34/hr 72.49
i5-750-4gb Intel i5 750 2.67 4 cores 4 2 SATA Raid 1 220GB 0.206/hr 69.47
storm-4gb Intel i5 750 2.67 2 cores 3.5 300 GB 0.14/hr 60.86
i5-750-2gb Intel i5 750 2.67 4 cores 2 1 200GB SATA No Raid 0.171/hr 49.76
e5506x2-4gb Xeon E5506 2.13 8 cores 8 1 453GB SATA No Raid 0.274/hr 48.37
e5506x2-4gb Xeon E5506 2.13 8 cores 8 2 SATA Raid 1 453GB 0.322/hr 46.8
storm-2gb Intel Q9400 2.66 1 core 1.7 150 GB 0.07/hr 34.95


GoGrid
GoGrid's disk IO performance was excellent across instances of all sizes. Even the low end 1GB cloud server scored almost 10% faster than the baseline. The 4GB cloud server was the #2 performer in this post at over 60% better performance than the baseline.
GoGrid's cloud servers run off of local host storage. They offer prepaid plans starting at $199/mo which discounts usage to $0.08/GB/hr (versus non-prepaid $0.19/GB/hr).

GoGrid [CA, US]
ID CPU Memory Storage Price IOP
gg-4gb Xeon E5520 2.27 4 cores 4 240 GB 0.76/hr 161.14
gg-2gb Xeon E5520 2.27 2 cores 2 120 GB 0.38/hr 133.61
gg-8gb Xeon E5450 2.99 6 cores 8 480 GB 1.52/hr 130.68
gg-1gb Xeon E5520 2.27 1 core 1 60 GB 0.19/hr 108.19


Voxel
Voxel's cloud server storage resides on external SANs in each data center. Disk IO performance was roughly 50% of the baseline across the board, about average for providers using external instance storage. The exact technical details of Voxel's SAN storage are not available.

Voxel [New York]
ID CPU Memory Storage Price IOP
vx-4gb-ny Xeon L5520 2.26 2 cores 4 40 GB 0.211/hr 49.65
vx-2gb-ny Xeon L5520 2.26 1 core 2 20 GB 0.106/hr 41.9

Voxel [Amsterdam]
ID CPU Memory Storage Price IOP
vx-2gb-nl Xeon L5520 2.26 1 core 2 20 GB 0.106/hr 40.61

Voxel [Singapore]
ID CPU Memory Storage Price IOP
vx-4gb-sg Xeon L5520 2.26 2 cores 4 40 GB 0.211/hr 51.52
vx-2gb-sg Xeon L5520 2.26 1 core 2 20 GB 0.106/hr 43.49

NewServers
NewServers offers a true bare metal cloud. When you create a server instance it actually deploys onto physical hardware including local disk storage (no hypervisor overhead). Like the baseline server, the ns-fast instance below uses SAS storage. According to NewServers, the SAS drives used in this instance are Seagate 10K RPM. The slower spindle speed and Raid 1 configuration may have attributed to the reduced performance of that server compared with the baseline.

NewServers [FL, US]
ID CPU Memory Storage Price IOP
ns-jumbo Xeon E5504 2.00 8 cores 48 2 x 500GB SATA RAID 1 0.6/hr 84.15
ns-fast Xeon E5450 2.99 4 cores 4 2 x 300GB SAS RAID 1 0.53/hr 51.97
ns-large Xeon E5405 2.00 4 cores 4 1 x 250GB SATA 0.25/hr 41.17
ns-med Opteron 3.20 2 cores 2 2 x 73GB SCSI Raid 1 0.17/hr 29.42
ns-small Opteron 2.80 1 core 1 1 x 36GB SCSI 0.11/hr 22.79


Linode
Linode markets itself as a VPS provider. However, because they are a very common and popular provider, and support some cloud-like features, we included them in our benchmarks. Linode uses Raid 1 local storage. The higher end servers generally performed better. The ln-14400 performed slightly better than the baseline server.

Linode VPS Hosting [Atlanta]
ID CPU Memory Storage Price IOP
ln-14400 Xeon L5520 2.27 4 cores 14.06 640 GB 26.67/day 106.06
ln-5760 Xeon L5520 2.27 4 cores 5.54 256 GB 10.67/day 96.21
ln-8640 Xeon L5520 2.27 4 cores 8.45 383 GB 16/day 95.38
ln-2880 Xeon L5520 2.27 4 cores 2.81 128 GB 5.33/day 47.17
ln-1080 Xeon L5520 2.27 4 cores 1.05 48 GB 2/day 34.47


SoftLayer
SoftLayer's cloud server storage resides on external SANs. Disk IO performance ranged from OK to painfully slow and showed a general increase in performance on larger sized instances. We believe storage is based on a GigE iSCCI SAN.

SoftLayer [Dallas]
ID CPU Memory Storage Price IOP
sl-2gb-dallas Xeon X3460 2.80 2 cores 2 100 GB 0.25/hr 41.07
sl-4gb-dallas Xeon E5520 2.27 4 cores 4 100 GB 0.35/hr 31.22
sl-1gb-dallas Xeon X3460 2.80 1 core 1 100 GB 0.15/hr 22.6


SoftLayer [WDC]
ID CPU Memory Storage Price IOP
sl-4gb-wdc Xeon X3460 2.80 4 cores 4 100 GB 0.35/hr 51.07
sl-2gb-wdc Xeon X3460 2.80 2 cores 2 100 GB 0.25/hr 41.53
sl-1gb-wdc Xeon X3460 2.80 1 core 1 100 GB 0.15/hr 19.33


SoftLayer [Seattle]
ID CPU Memory Storage Price IOP
sl-4gb-seattle Xeon X3460 2.80 4 cores 4 100 GB 0.35/hr 49.49
sl-2gb-seattle Xeon X3460 2.80 2 cores 2 100 GB 0.25/hr 39.74
sl-1gb-seattle Xeon X3460 2.80 1 core 1 100 GB 0.15/hr 24.89


Terremark
Terremark is a VMWare vCloud provider. Terremark was the best performing provider in this post using external SAN storage and providing automatic failover capabilities. Use of the SAN and VMWare provide a high availability feature wherein servers are automatically migrated to another host should the existing host fail.

Terremark vCloud Express [FL, US]
ID CPU Memory Storage Price IOP
tm-8gb-4vpu Opteron 8389 2.91 4 cores 8 80 GB 0.64/hr 117.5
tm-16gb-8vpu Opteron 8389 2.91 8 cores 16 160 GB 1.672/hr 101.85
tm-4gb-2vpu Opteron 8389 2.91 2 cores 4 40 GB 0.305/hr 95.86
tm-2gb Opteron 8389 2.91 1 core 2 20 GB 0.137/hr 88.99
tm-1gb Opteron 8389 2.91 1 core 1 10 GB 0.074/hr 74.68


OpSource Cloud
OpSource is another VMWare-based provider. C
loud server storage resides on an external SAN. Performance was generally slower than other SAN/VMWare providers in this post. Performance was roughly flat across instances of all sizes.

OpSource Cloud [VA, US]
ID CPU Memory Storage Price IOP
os-2gb Xeon X7460 2.66 1 core 2 20 GB 0.296/hr 42.98
os-16gb-4cpu Xeon X7460 2.66 4 cores 16 160 GB 0.808/hr 40.84
os-32gb-4cpu Xeon X7460 2.66 4 cores 32 320 GB 1.256/hr 40.73
os-8gb-4cpu Xeon X7460 2.66 4 cores 8 80 GB 0.584/hr 39.75
os-4gb-2cpu Xeon X7460 2.66 2 cores 4 40 GB 0.392/hr 38.24
os-1gb Xeon X7460 2.66 1 core 1 10 GB 0.268/hr 37.34


Speedyrails
Speedyrails is a VPS provider based out of Quebec Canada. Cloud server storage is local to the host.

Speedyrails [QC, CA]
ID CPU Memory Storage Price IOP
sr-4gb Xeon E5520 2.27 8 cores 4 160 GB 8/day 66.72
sr-2gb Xeon E5520 2.27 8 cores 2 80 GB 4.27/day 57.99
sr-1gb Xeon E5520 2.27 8 cores 1 40 GB 2.27/day 33.43


Zerigo
Zerigo is a VPS and Cloud Server vendor based out of Denver, CO.
Cloud server storage is local to the host.

Zerigo [CO, US]
ID CPU Memory Storage Price IOP
zr-4gb Opteron 2374 2.20 4 cores 4 160 GB 0.24/hr 52.64
zr-2gb Opteron 2374 2.20 4 cores 2 80 GB 0.12/hr 42.2
zr-1gb Opteron 2374 2.20 4 cores 1 40 GB 0.06/hr 29.12


ReliaCloud
ReliaCloud uses an external SAN for cloud server instance storage. Average performance was slower than other external storage providers.

ReliaCloud Cloud Services [MN, US]
ID CPU Memory Storage Price IOP
rc-8gb 8 800 GB 0.64/hr 29.82
rc-2gb Xeon E5504 4.40 2 cores 2 200 GB 0.16/hr 27.41
rc-1gb Xeon E5504 2.00 1 core 1 100 GB 0.08/hr 23.83
rc-4gb Xeon E5504 2.00 4 cores 4 400 GB 0.32/hr 23.32


IBM Development & Test Cloud
IBM Developer Cloud also uses SAN storage. Disk IO performance was about average compared with other external storage providers.

IBM Development & Test Cloud [NY, US]
ID CPU Memory Storage Price IOP
ibm-dev-large Xeon X5570 2.93 8 cores 3.5 30 GB 56.53
ibm-dev-med Xeon X5570 2.93 4 cores 1.75 20 GB 51.82


BlueLock
BlueLock is a VMWare vCloud provider. Server storage resides on a SAN.
Combined SAN and VMWare provide a high availability feature wherein servers are automatically migrated to another host should the existing host fail. Performance was very good for external storage providers and generally showed a linear increase in performance on larger sized instances.

BlueLock [IN, US]
ID CPU Memory Storage Price IOP
bl-8gb-4cpu Xeon X5550 2.67 4 cores 8 160 GB 0.661/hr 71.68
bl-16gb-8cpu Xeon X5550 2.67 8 cores 16 320 GB 1.729/hr 62.49
bl-4gb-2cpu Xeon X5550 2.67 2 cores 4 80 GB 0.308/hr 54
bl-2gb Xeon X5550 2.67 1 core 2 40 GB 0.134/hr 41.35
bl-1gb Xeon X5550 2.67 1 core 1 20 GB 0.068/hr 25.75


CloudCentral
Cloud Central is a new cloud server provider in Australia. Cloud Central uses SAN storage. Performance was very good compared with other SAN providers.

Cloud Central [AU]
ID CPU Memory Storage Price (AUD) IOP
cc-large Opteron 2374 2.20 4 cores 8 512 GB 0.96/hr 77.11
cc-huge Opteron 2374 2.20 4 cores 16 1024 GB 1.92/hr 73.77
cc-reg Opteron 2374 2.20 4 cores 4 256 GB 0.48/hr 68.53
cc-med Opteron 2374 2.20 4 cores 2 128 GB 0.24/hr 62.42
cc-small Opteron 2374 2.20 4 cores 1 64 GB 0.12/hr 44.27


RimuHosting is a VPS provider based out of New Zealand. Cloud server storage is local to the host.

RimuHosting [NZ]
ID CPU Memory Storage Price (NZD) IOP
rh-nz-2gb Xeon E5506 2.13 2 cores 2 16 GB 210.88/mo 38.47


ElasticHosts is a UK cloud provider. We are unaware if storage is local or external.

ElasticHosts [UK]
ID CPU Memory Storage Price (GBP) IOP
eh-8gb-20gh Xeon E5420 2.50 6 cores 8 80 GB 0.654/hr 57.91
eh-4gb-8gh Xeon E5420 2.50 4 cores 4 40 GB 0.326/hr 33.06
eh-2gb-4gh Xeon E5420 2.50 2 cores 2 20 GB 0.164/hr 31.25
eh-1gb-2gh Xeon E5420 2.50 1 core 1 10 GB 0.082/hr 23.1


Flexiscale
Flexiscale is a UK cloud provider that has been around for a few years. They were recently acquired and renamed to Flexiant. They have recently released version 2.0 of their cloud server platform. They use external SAN storage.

8/17/2010 Update: Since originally conducting these tests in May 2010, Flexiscale has updated their external storage system. The top 8gb instance in the table below represents benchmark results conducted after than update. These results show a dramatic increase in IO performance (about 35%).


Flexiscale [UK]
ID CPU Memory Storage IOP
8gb AMD Opteron 8356 2.29 GHz [1 processor, 4 cores] 8 320 GB 84.93
8gb AMD Opteron 8218 2.62 GHz [1 processor, 4 cores] 8 320 GB 62.71
4gb AMD Opteron 8220 2.80 GHz [1 processor, 2 cores] 8 160 GB 41.77
2gb AMD Opteron 8218 2.60 GHz [1 processor, 1 core] 8 80 GB 36.56
1gb AMD Opteron 8218 2.60 GHz [1 processor, 1 core] 8 40 GB 19.46



Summary
This is our first attempt at comparing cloud provider disk IO performance. We acknowledge that it is not perfect and hope to make improvements over time. Please comment on this post if you have any suggestions on how we might improve our methods. We intend to continually run these benchmarks (every couple of months) to incorporate new providers, improve the quality and quantity of our data available, and check for provider upgrades and improvements.

Providers were about evenly split between those using external SAN storage versus those using local host storage for cloud server instance storage. Generally the local storage providers performed better in the benchmarks. However, most of the external SAN providers are able to incorporate better durability including implicit backups and automatic host server failover.