Best Amazon EC2 instance type for Ray Cloud Browser on metagenomes and bacterial genomes

-
Hi,

I am using spot instances on Amazon Elastic Compute Cloud (EC2) to deploy a few installations of Ray Cloud Browser. Initially, I opted for m1.small instances because the Ray Cloud Browser web service that answers a bunch of HTTP GET API calls was not optimized. Namely, the C++ back-end code was memory-mapping a huge file (~16-20 GiB). This huge bloated binary file was the index -- the source of information about the huge graph describing a given biological sample. A recent patch improved the performance by packing information in every available bit in the binary file, reducing the number of blocks by 75%, hereby enhancing performance as well.

Recently, there were a lot of peaks in the m1.small spot pricing, and I figured out that my use case was all about bursts -- discrete HTTP API calls.


I then looked at the pricing history for the last 3 months, and these pesky peaks seem to be a 2013 thing.

The t1.micro spot instance pricing history also has these sophisticated highs.

With all these recent pricing surges, I decided to provision some standard Elastic Block Storage (EBS) volumes so that my stuff stays in the cloud when the spot market price exceeds what I want to pay as a customer.

Before today, I was using ephemeral volumes, which are basically just EBS storage volumes that vanish when the instance is stopped or terminated.
I don't use ephemeral EBS volumes anymore as they are not useful for my user story.

Running one m1.small on-demand instance costs 47.48 $ per month whereas one single 64-GiB EBS standard volume costs 6.40 $ per month (excluding input/output requests). And running one m1.small spot instance costs around 5.11 $ per month whereas running one t1.micro spot instance costs around 2.19 $ per month.

Therefore, running Ray Cloud Browser on one t1.micro spot instance using one 64-GiB EBS volume costs 8.59 $ per month. Basically, it costs nothing at all considering the cost of other things in genomics research, such as sequencing runs on instruments, bioinformatician-hours, developper-hours, and so on.

And the software called Ray Cloud Browser costs nothing too -- I authored it and it's free software distributed under the GNU General Public License version 3.

I hope that people everywhere use Ray Cloud Browser to visualize their DNA samples !

The cloud is really a nice thing, it removes barriers. But you have to use it to gain experience in order to lower your costs.


Comments

Post a Comment

Popular posts from this blog

A survey of the burgeoning industry of cloud genomics

-
Image
I like lists. Here is a list of companies in the cloud genomics industry. Table 1: Companies acting in the industry of cloud genomics. The list is in no particular order. Company Products People Founded Link GenoSpace, LLC - GenoSpace platform - John Quackenbush (CEO) 2011 https://www.genospace.com/ DNAnexus, Inc. - DNAnexus Platform - DNAnexus Platform SDK (a.k.a. dx-toolkit) - Serafim Batzoglou (co-founder) - Andreas Sundquist (co-founder and CTO) - a bunch of other famous people in the field 2009 https://dnanexus.com/ Seven Bridges Genomics, Inc. - IGOR - IGOR Python SDK - Igor Bogicevic (CTO) 2009 https://www.sbgenomics.com/ BGI - EasyGenomics - Xing Xu (Senior Product Manager) Introduced in 2012 https://www.easygenomics.com/ Illumina, Inc. - BaseSpace™ - BaseSpace API - Alex Dickinson (SVP of Cloud Genomics) Introduced in 2011 https://basespace.illumina.com/ Era7 Information Technologies SLU - BG7...
Read more

Generating neural machine instructions for multi-head attention

-
  An animal can do training and inference every day of its existence until the day of its death. A forward pass is all you need.   Here is the current model configuration that I use for multi-head attention:         let sequence_length = 6;         let vocab_size = 20;         let n_embd = 64;         let num_heads = 8; You can see the full model here: https://github.com/sebhtml/novigrad/blob/8df6192829edd9cda3a221e12affa770fbda2a1b/src/models/mega_man_attention.rs The struct uses Embedding, MultiHeadAttention, Linear, Softmax: struct MegaManAttentionModel {     input_shape: Vec<usize>,     output_shape: Vec<usize>,     vocab_size: usize,     sequence_length: usize,     embedding: Embedding,     multi_head_attention: MultiHeadAttention, ...
Read more

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

-
On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands: Install FreeBSD on a 10G disk. vm create -s 10G guestname   vm install FreeBSD-12.1-RELEASE-amd64-memstick.img vm console guestname Add 8 30GB zvol-based disks vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname You will need to do vm poweroff guestname vm start guestname because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work) *** If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template: root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145 root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145 r...
Read more