Measuring latency in the cloud

Hey,

I started 10 cc2.8xlarge in Amazon EC2 using the spot instance market.

The cc2.8xlarge hourly rate sits at 0.270 $ / h. The on-demand rate is 2.400 $ / h.

Results

Table 1: Cloud latencies. Number of instances was 10 and the instance type was cc2.8xlarge.

MPI ranks
 MPI ranks per instance
 Average roundtrip latency (microseconds)
10
 1
 235.200785
20
 2
 313.960899
40
 4
 365.403384
80
 8
 473.863469
120
 12
 563.779653
160
 16
 322.942884
200
 20
 258.164757
250
 25
 220.151894
320
 32
 280.532563


Instances


The specification of 1 cc2.8xlarge is:

Cluster Compute Eight Extra Large Instance

60.5 GiB of memory
88 EC2 Compute Units (2 x Intel Xeon E5-2670, eight-core)
3370 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
EBS-Optimized Available: No*
API name: cc2.8xlarge


(that's 32 VCPUs with hyperthreading !)

These are the 10 instances in my placement group:

i-b078b9c0: ec2-174-129-96-90.compute-1.amazonaws.com
i-ba78b9ca: ec2-54-234-6-223.compute-1.amazonaws.com
i-b278b9c2: ec2-107-21-145-1.compute-1.amazonaws.com
i-b678b9c6: ec2-204-236-254-68.compute-1.amazonaws.com
i-b878b9c8: ec2-50-16-171-238.compute-1.amazonaws.com
i-bc78b9cc: ec2-54-234-75-41.compute-1.amazonaws.com
i-a678b9d6: ec2-23-22-107-180.compute-1.amazonaws.com
i-a878b9d8: ec2-23-22-77-54.compute-1.amazonaws.com
i-aa78b9da: ec2-23-21-6-134.compute-1.amazonaws.com
i-9678b9e6: ec2-50-16-135-211.compute-1.amazonaws.com

Software used to measuring latency


I will test the latency using latency_checker.

Latency checker uses a any-to-any communication pattern and the default message size is 4000 bytes (excluding envelope).

First, we have to setup keys.

I then tested the 10 links:

[ec2-user@ip-10-156-160-75 ~]$ cat hosts.txt
ec2-174-129-96-90.compute-1.amazonaws.com
ec2-54-234-6-223.compute-1.amazonaws.com
ec2-107-21-145-1.compute-1.amazonaws.com
ec2-204-236-254-68.compute-1.amazonaws.com
ec2-50-16-171-238.compute-1.amazonaws.com
ec2-54-234-75-41.compute-1.amazonaws.com
ec2-23-22-107-180.compute-1.amazonaws.com
ec2-23-22-77-54.compute-1.amazonaws.com
ec2-23-21-6-134.compute-1.amazonaws.com
ec2-50-16-135-211.compute-1.amazonaws.com

[ec2-user@ip-10-156-160-75 ~]$  for i in $(cat hosts.txt ); do ssh $i hostname; done
ip-10-156-160-75
ip-10-156-210-34
ip-10-156-164-15
ip-10-156-210-142
ip-10-156-209-237
ip-10-156-210-243
ip-10-156-160-214
ip-10-156-161-58
ip-10-156-224-53
ip-10-156-225-18

That's great, let's install packages on each instance:

[ec2-user@ip-10-156-160-75 ~]$ sudo yum install -y git openmpi openmpi-devel gcc make; exit

[ec2-user@ip-10-156-160-75 ~]$ mkdir software; cd software
[ec2-user@ip-10-156-160-75 ~]$ git clone git://github.com/sebhtml/latency_checker.git
Cloning into latency_checker...
remote: Counting objects: 51, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 51 (delta 27), reused 49 (delta 25)
Receiving objects: 100% (51/51), 25.14 KiB, done.
Resolving deltas: 100% (27/27), done.

[ec2-user@ip-10-156-160-75 ~]$ cd latency_checker/
[ec2-user@ip-10-156-160-75 latency_checker]$ load_openmpi
[ec2-user@ip-10-156-160-75 latency_checker]$ make
mpicc -O3 -Wall -ansi -DASSERT -c main.c -o main.o
mpicc -O3 -Wall -ansi -DASSERT -c process.c -o process.o
mpicc -O3 -Wall -ansi -DASSERT main.o process.o -o latency_checker

The last step before the actual tests is to copy the executable on all 10 instances.

[ec2-user@ip-10-156-160-75 ~]$ for i in $(cat hosts.txt ); do scp software/latency_checker/latency_checker $i:; done


Measuring latency


[ec2-user@ip-10-156-160-75 ~]$ /usr/lib64/openmpi/bin/mpiexec -hostfile hosts.txt -n 10 ./latency_checker -exchanges 10000 -message-size 4000|grep "average of average roundtrip latencies"
Rank 0 -> average of average roundtrip latencies: 235.200785 microseconds

Software versions:

latency_checker v1.0.0
gcc (GCC) 4.6.2 20111027 (Red Hat 4.6.2-2)
openmpi 1.5.4 mockbuild@gobi-build-31004.sea31.amazon.com
Linux 3.2.30-49.59.amzn1.x86_64

Comments

Unknown said…
This is measuring latency between all 10 instances and themselves right? Are they all in the same availability zone? Seems pretty high numbers if that's the case.
Friday, January 25, 2013 at 10:43:00 PM EST
sebhtml said…
Hey,

> This is measuring latency between
> all 10 instances and themselves
> right?

Yes. However there are more than 1 MPI rank on each instance, and each MPI rank talks to all the others including itself (see table).

This any-to-any communication pattern is known to be a bad design, so it's a good test to stress instances.

For my research project, I route my messages using a polytope to avoid the insane any-to-any communication pattern except when I am using good interconnects such as Cray XE6, Blue Gene/Q, or Intel QLogic.

Finally, the messages here are not empty shells -- they are 4000-byte messages in both direction.

The latency is for roundtrip transit, so you have to divide by 2 to get the real thing I suppose.

> Are they all in the same availability zone?

I assume they were because I in my spot requests I provided a placement group. But hey, all I know is that it's running in the cloud.

> Seems pretty high numbers if that's the case.

Yes, I think that these are high numbers too. Maybe I did something wrong.

What numbers were you expecting ?
Monday, January 28, 2013 at 10:08:00 AM EST
Post a Comment

Popular posts from this blog

My 2022 Calisthenics split routine

Hey my fellow blog readers ! At the beginning of year 2022, I watched some YouTube videos from Tom Merrick, The Bodyweight Warrior . The video 3 Things I Wish I Knew When I Started made me think a lot. Since I began my bodyweight journey in July 2020, I used a full-body routine, 3 times per week. As I became stronger, the volume of strength work became too high. Watching videos from Tom Merrick convinced me to split my full body routine into a split routine. My new Calisthenics routine is a split routine. Splitting the routine allowed me to increase rest time between upper body workouts, and also to increase rest time between lower body workouts. It also allowed me to increase from 3 (full-body) workouts per week to 4 workouts (upper body, lower body + core, upper body, lower body + core) per week. Here what my split routine looks like: Wednesday: Upper Body Workout (1h15min) Pair 1 Ring Pull-Up Pseudo Planche Push-Up  Pair 2 Tuck Assisted Victorian Box Handstand Push-Up Pair 3 Ring R
Read more

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands: Install FreeBSD on a 10G disk. vm create -s 10G guestname   vm install FreeBSD-12.1-RELEASE-amd64-memstick.img vm console guestname Add 8 30GB zvol-based disks vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname You will need to do vm poweroff guestname vm start guestname because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work) *** If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template: root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145 root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145 r
Read more

Le tissu adipeux brun, la thermogénèse, et les bains froids

Image
Figure 4 de Mapping of human brown adipose tissue in lean and obese young men https://www.pnas.org/content/114/32/8649 BAT = Brown Adipose Tissue    Avec l'hiver à nos portes, je partage ci-dessous mon intérêt pour le tissu adipeux brun. Le tissu adipeux blanc a un rôle d'isolant et de stockage. Le tissu adipeux brun ("gras brun") diffère du traditionnel tissue adipeux blanc. Le tissus adipeux brun a un rôle de plinthe électrique (thermogénèse). Et non, manger de la galvaude ne donne pas plus de gras brun.😄 Selon le livre Human Anatomy & Physiology de Élaine Marieb , 5ème édition, publié en 2001, le tissu adipeux brun est présent seulement chez les bébés, qui ne peuvent pas frissonner. Le tissu adipeux brun fonctionnel a été découvert chez les adultes en 2009 ( New England Journal of Medicine ). Le tissu adipeux brun a été cartographié chez l'humain adulte en 2017 ( Proceedings of the National Academy of Sciences ). J'ai mis la Figure 4 de l'articl
Read more