1 human genome, 1 Amazon EC2 instance, x days

-

Today we are assembling trying to assemble 1 human genome on 1 Amazon EC2 instance using SSD disks. We pay by the hour, 3.1 $ per hour to be exact.
 

Colors


black -> just text explaining stuff
purple -> commands to type on your laptop as a normal user with ec2-api-tools configured (I use a Lenovo X230 with Fedora)
red -> commands to type on the instance as root
blue -> commands to type on the instance as ec2-user

High I/O Quadruple Extra Large Instance

 
60.5 GiB of memory
35 EC2 Compute Units (16 virtual cores*)
2 SSD-based volumes each with 1024 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
Storage I/O Performance: Very High**
EBS-Optimized Available: No***
API name: hi1.4xlarge

Blog post

Pricing (on-demand for Linux/UNIX usage)


$3.100 per Hour


Starting the instance


1..2..3.. let's start!

$ ec2-authorize default -p 22

$ ec2-run-instances ami-08249861 -t hi1.4xlarge -n 1 -b /dev/sdf=ephemeral0 -b /dev/sdg=ephemeral1 -k GoldThor -g default

$ ec2-describe-instances |grep INS|grep hi1.4xlarge|grep -v terminated
INSTANCE    i-a7e6d4d6    ami-08249861    ec2-23-22-97-73.compute-1.amazonaws.com    ip-10-148-132-199.ec2.internal    running    GoldThor    0    hi1.4xlarge    2013-01-18T18:54:44+0000    us-east-1c                monitoring-disabled    23.22.97.73    10.148.132.199        ebs                    hvm    xen        sg-6aa44c02    default    false   

Connecting to the cloud


$ ssh -i GoldThor.pem  ec2-user@ec2-23-22-97-73.compute-1.amazonaws.com


Using SSD disks for virtual memory


[root@ip-10-148-132-199 ec2-user]# mkswap /dev/xvdf  -f
Setting up swapspace version 1, size = 1073741820 KiB
no label, UUID=b1cc9a0e-3fdb-47b3-bdde-3d0fdfadcf20

[root@ip-10-148-132-199 ec2-user]# mkswap /dev/xvdg  -f
Setting up swapspace version 1, size = 1073741820 KiB
no label, UUID=f452b610-80ea-4b5d-b3a9-751ba1e32f55

[root@ip-10-148-132-199 ec2-user]# swapon /dev/xvdf

[root@ip-10-148-132-199 ec2-user]# swapon /dev/xvdg

[ec2-user@ip-10-148-132-199 ~]$ grep Total /proc/meminfo |head -n2
MemTotal:       61370008 kB
SwapTotal:      2147483640 kB

Finding a cool dataset


$ ec2-describe-snapshots -a |grep Illumina|grep Jay|grep Linux
SNAPSHOT    snap-53b3cb3a    vol-ce0dcba7    completed    2010-01-10T20:48:42+0000    100%    947081328633    350    Illumina - Jay Flatley Genome Data Set (Linux)

Here, I used provisioned, but it does not matter because the snapshot takes a while anyway to transit from Amazon S3 to Amazon EBS.

$ ec2-create-volume --snapshot  snap-53b3cb3a -z  us-east-1c -t io1  --iops 2000
VOLUME    vol-c2d096bc    350    snap-53b3cb3a    us-east-1c    creating    2013-01-18T21:01:07+0000    io1    2000

$ ec2-attach-volume vol-c2d096bc -i   i-a7e6d4d6   -d /dev/sdx
ATTACHMENT    vol-c2d096bc    i-a7e6d4d6    /dev/sdx    attaching    2013-01-18T21:02:25+0000   

[root@ip-10-148-132-199 ec2-user]# mkdir /mnt/dataset
[root@ip-10-148-132-199 ec2-user]# mount /dev/sdx /mnt/dataset/

[root@ip-10-148-132-199 ~]# chown ec2-user /mnt/dataset/Alignments -R


Now we need to touch all the blocks otherwise Ray will just be waiting for transit from S3 to EBS because EBS is copy-on-write, if you don't use a block, then it will eventually be there, but it can take a while because it out-of-order (or should I say it's the order the file system ext3 decides to touch blocks).


[root@ip-10-148-132-199 ~]# dd if=/dev/sdx of=/dev/null bs=10M

(dd took around 1 hour, 52 minutes, rate was 42.7 MiB/s)

It would be more cost-effective to do the dd command from a cheaper instance type.

Installing analysis software


[root@ip-10-148-132-199 ec2-user]# yum install git gcc-c++ make openmpi-devel openmpi


[ec2-user@ip-10-148-132-199 ~]$ git clone git://github.com/sebhtml/ray.git
Cloning into ray...
remote: Counting objects: 21530, done.
remote: Compressing objects: 100% (4255/4255), done.
remote: Total 21530 (delta 17534), reused 21153 (delta 17179)
Receiving objects: 100% (21530/21530), 21.47 MiB | 13.59 MiB/s, done.
Resolving deltas: 100% (17534/17534), done.

[ec2-user@ip-10-148-132-199 ~]$ git clone git://github.com/sebhtml/RayPlatform.git
Cloning into RayPlatform...
remote: Counting objects: 17903, done.
remote: Compressing objects: 100% (3325/3325), done.
remote: Total 17903 (delta 14535), reused 17872 (delta 14504)
Receiving objects: 100% (17903/17903), 20.75 MiB | 4.88 MiB/s, done.
Resolving deltas: 100% (14535/14535), done.

[ec2-user@ip-10-148-132-199 ~]$ cd ray

[ec2-user@ip-10-148-132-199 ray]$ load_openmpi

[ec2-user@ip-10-148-132-199 ray]$ make -j 16 PREFIX=/home/ec2-user/software/ray/v-e2c32ffdbe5963b ASSERT=y CXXFLAGS="-Wall -std=c++98 -O3 -march=native"

[ec2-user@ip-10-148-132-199 ray]$ make install

Installing Ray to /home/ec2-user/software/ray/v-e2c32ffdbe5963b

[ec2-user@ip-10-148-132-199 ray]$ mpiexec  -n 1 /home/ec2-user/software/ray/v-e2c32ffdbe5963b/Ray -version
Ray version 2.1.1-devel
License for Ray: GNU General Public License version 3
RayPlatform version: 1.2.0-devel
License for RayPlatform: GNU Lesser General Public License version 3

MAXKMERLENGTH: 32
Maximum coverage depth stored by CoverageDepth: 4294967295
MAXIMUM_MESSAGE_SIZE_IN_BYTES: 4000 bytes
FORCE_PACKING = n
ASSERT = y
HAVE_LIBZ = n
HAVE_LIBBZ2 = n
PROFILER_COLLECT = n
CLOCK_GETTIME = n
__linux__ = y
_MSC_VER = n
__GNUC__ = y
RAY_32_BITS = n
RAY_64_BITS = y
MPI standard version: MPI 2.1
MPI library: Open-MPI 1.5.4
Compiler: GNU gcc/g++ 4.6.2 20111027 (Red Hat 4.6.2-2)
With SSE 4.2
With hardware pop count

Preparing the job file


[ec2-user@ip-10-148-132-199 ~]$ cat create-job.sh
for i in $(find /mnt/dataset/|grep export.txt$|grep _1.export)
do
        echo "-p \\"
        echo "       $i \\"
        echo "       $(echo $i|sed 's/_1_export/_2_export/g') \\"
done

[ec2-user@ip-10-148-132-199 ~]$ bash create-job.sh  > Run-Ray-on-SSDs.sh


Edit the file to get this:

[ec2-user@ip-10-148-132-199 ~]$ head Run-Ray-on-SSDs.sh
#!/bin/bash

. /etc/profile.d/openmpi.sh
load_openmpi
PATH=/home/ec2-user/software/ray/v-e2c32ffdbe5963b/:$PATH

mpiexec -n 16 Ray -k 31 -o Human-Genome-EC2-2013-01-18-1 \
-p \
       /mnt/dataset/Alignments/090416_EAS013R_0019_FC3159L/s_6_1_export.txt \
       /mnt/dataset/Alignments/090416_EAS013R_0019_FC3159L/s_6_2_export.txt \

 [ec2-user@ip-10-148-132-199 ~]$ chmod +x Run-Ray-on-SSDs.sh


Launching the job


[ec2-user@ip-10-148-132-199 ~]$ ./Run-Ray-on-SSDs.sh  &>  Human-Genome-EC2-2013-01-18-1.log


Update:

Basically, Ray loaded sequences, started to build the graph. But the 16 processes became slow when it started to swap heavily.

It turns out that context switches for swapping memory pages is killing performance after all.

[ec2-user@ip-10-148-132-199 ~]$ tail /proc/28585/status -n2
voluntary_ctxt_switches:        6625309
nonvoluntary_ctxt_switches:     10787012

Regardless, this was a fun experience.

[ec2-user@ip-10-148-132-199 ~]$ top -n 1
Tasks: 142 total,  15 running, 127 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.4%us,  8.7%sy,  0.0%ni, 71.2%id,  8.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  61370008k total, 61057092k used,   312916k free,     3080k buffers
Swap: 2147483640k total, 121548288k used, 2025935352k free,    13840k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                  
28584 ec2-user  20   0 11.0g 3.5g 2224 R 100.0  5.9 219:28.68 Ray                                                                                      
  468 root      20   0     0    0    0 S 88.8  0.0 119:29.33 kswapd0                                                                                   
28583 ec2-user  20   0 11.0g 3.5g 2228 R 86.8  5.9 225:19.30 Ray                                                                                       
28575 ec2-user  20   0 11.0g 3.5g 2228 R 84.8  5.9 208:18.21 Ray                                                                                       
28581 ec2-user  20   0 10.7g 3.5g 2224 R 78.9  5.9 213:03.02 Ray                                                                                       
28586 ec2-user  20   0 11.0g 3.5g 2216 R 78.9  5.9 260:32.81 Ray                                                                                       
28580 ec2-user  20   0 10.8g 3.5g 2224 R 77.0  5.9 216:27.89 Ray                                                                                       
28577 ec2-user  20   0 11.0g 3.5g 2224 R 69.1  5.9 208:44.33 Ray                                                                                       
28571 ec2-user  20   0 11.0g 3.5g 2240 R 63.1  5.9 269:15.16 Ray                                                                                       
28573 ec2-user  20   0 10.9g 3.5g 2228 R 63.1  5.9 207:47.91 Ray                                                                                       
28576 ec2-user  20   0 11.0g 3.5g 2228 D 59.2  5.9 208:28.14 Ray                                                                                       
28579 ec2-user  20   0 11.0g 3.5g 2224 R 53.3  5.9 216:33.98 Ray                                                                                       
28582 ec2-user  20   0 11.0g 3.5g 2224 R 49.3  5.9 214:08.51 Ray                                                                                       
28572 ec2-user  20   0 10.9g 3.5g 2228 R 37.5  5.9 285:00.21 Ray                                                                                       
28578 ec2-user  20   0 11.0g 3.5g 2224 R 27.6  5.9 210:39.44 Ray                                                                                       
28574 ec2-user  20   0 11.0g 3.5g 2228 R 25.7  5.9 206:08.48 Ray                                                                                       
28585 ec2-user  20   0 11.0g 3.5g 2224 D 19.7  5.9 229:11.74 Ray                                                                                       
15519 ec2-user  20   0 15068 1204  888 R  2.0  0.0   0:00.01 top                                                                                       
    1 root      20   0 19400  200  200 S  0.0  0.0   0:02.20 init                                                                                      
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                  
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.90 ksoftirqd/0                                                                               
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0                                                                               
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.05 kworker/u:0                                                                               
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                               
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                                               
    8 root      20   0     0    0    0 S  0.0  0.0   0:01.44 kworker/1:0                                                                               
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.24 ksoftirqd/1                                                                               
   10 root      20   0     0    0    0 S  0.0  0.0   0:06.15 kworker/0:1

Comments

Anonymous said…
Would appreciate if you could give a step by step similar to this post on how to launch on the best EC instance that utilises RAM for running Ray.

Will the steps be similar if I wanted to run it on CloudBioLinux?

I wonder whether the distributed memory has impact on the performance on Ray assembler?
Tuesday, March 5, 2013 at 12:30:00 AM EST
sebhtml said…
> Would appreciate if you could give a step by step similar to this post on how to launch on the best EC instance that utilises RAM for running Ray.

The best EC2 instances for Ray are either cc2.8xlarge or the cr1.8xlarge.

> Will the steps be similar if I wanted to run it on CloudBioLinux?

I believe Ray is in CloudBioLinux (I submitted a patch which got merged).

> I wonder whether the distributed memory has impact on the performance on Ray assembler?

The impact is significant. That's why you must use placement groups so that your instances are not too far away from each other.
Tuesday, March 5, 2013 at 3:32:00 PM EST
Post a Comment

Popular posts from this blog

A survey of the burgeoning industry of cloud genomics

-
Image
I like lists. Here is a list of companies in the cloud genomics industry. Table 1: Companies acting in the industry of cloud genomics. The list is in no particular order. Company Products People Founded Link GenoSpace, LLC - GenoSpace platform - John Quackenbush (CEO) 2011 https://www.genospace.com/ DNAnexus, Inc. - DNAnexus Platform - DNAnexus Platform SDK (a.k.a. dx-toolkit) - Serafim Batzoglou (co-founder) - Andreas Sundquist (co-founder and CTO) - a bunch of other famous people in the field 2009 https://dnanexus.com/ Seven Bridges Genomics, Inc. - IGOR - IGOR Python SDK - Igor Bogicevic (CTO) 2009 https://www.sbgenomics.com/ BGI - EasyGenomics - Xing Xu (Senior Product Manager) Introduced in 2012 https://www.easygenomics.com/ Illumina, Inc. - BaseSpace™ - BaseSpace API - Alex Dickinson (SVP of Cloud Genomics) Introduced in 2011 https://basespace.illumina.com/ Era7 Information Technologies SLU - BG7...
Read more

Generating neural machine instructions for multi-head attention

-
  An animal can do training and inference every day of its existence until the day of its death. A forward pass is all you need.   Here is the current model configuration that I use for multi-head attention:         let sequence_length = 6;         let vocab_size = 20;         let n_embd = 64;         let num_heads = 8; You can see the full model here: https://github.com/sebhtml/novigrad/blob/8df6192829edd9cda3a221e12affa770fbda2a1b/src/models/mega_man_attention.rs The struct uses Embedding, MultiHeadAttention, Linear, Softmax: struct MegaManAttentionModel {     input_shape: Vec<usize>,     output_shape: Vec<usize>,     vocab_size: usize,     sequence_length: usize,     embedding: Embedding,     multi_head_attention: MultiHeadAttention, ...
Read more

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

-
On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands: Install FreeBSD on a 10G disk. vm create -s 10G guestname   vm install FreeBSD-12.1-RELEASE-amd64-memstick.img vm console guestname Add 8 30GB zvol-based disks vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname You will need to do vm poweroff guestname vm start guestname because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work) *** If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template: root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145 root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145 r...
Read more