Posts

Easing the programming of distributed systems

One-line summary: control messages are useful According to the paper entitled "Messaging Design Pattern and Pattern Implementation" , the business of sending a message involves four (4) participants:  Message Sender, Message Recipient (Receiver), Messenger, and Message. And also according to this document, a message can trigger a cascading effect. In distributed systems such as Dynamo , Cassandra , or other similar key-value stores, the storage system is distributed. And just like Lustre , clients are distributed too. A lot of use cases of these key-value storage systems is to prepare content for a customer, in general a web page or something like that. Once the content is prepared, it can be sent to the consumer and the process that prepared it can then discard its local copy of whatever was fetched from the key-value store. For the web, the client of these key-value stores (for example, the worker processes (or threads) of a web server) are not tightly coupled ...

A survey of the burgeoning industry of cloud genomics

Image
I like lists. Here is a list of companies in the cloud genomics industry. Table 1: Companies acting in the industry of cloud genomics. The list is in no particular order. Company Products People Founded Link GenoSpace, LLC - GenoSpace platform - John Quackenbush (CEO) 2011 https://www.genospace.com/ DNAnexus, Inc. - DNAnexus Platform - DNAnexus Platform SDK (a.k.a. dx-toolkit) - Serafim Batzoglou (co-founder) - Andreas Sundquist (co-founder and CTO) - a bunch of other famous people in the field 2009 https://dnanexus.com/ Seven Bridges Genomics, Inc. - IGOR - IGOR Python SDK - Igor Bogicevic (CTO) 2009 https://www.sbgenomics.com/ BGI - EasyGenomics - Xing Xu (Senior Product Manager) Introduced in 2012 https://www.easygenomics.com/ Illumina, Inc. - BaseSpace™ - BaseSpace API - Alex Dickinson (SVP of Cloud Genomics) Introduced in 2011 https://basespace.illumina.com/ Era7 Information Technologies SLU - BG7...

Spot instances that stop (or reboot) on their own: a new feature in AWS ?

So I have a spot instance for my visualization project . I am using one t1.micro instance in the spot market. My instance is i-05cabc6a and its spot request is sir-4e01da35 . It has a 64 GB EBS volume attached to it ( vol-0eba4a7f) and a 8 GB EBS volume for the operating system ( vol-b23d42e9).   According to the EC2 Management Console, its launch time was 2013-06-06 11:00 EDT (314 hours). This is a spot instance, it can not be stopped. It can only be terminated, or obviously it can continue to live too. So today I was showing people my visualization project, but it did not work. I use http://browser.cloud.raytrek.com/client for the permanent address. In the DNS: ;; ANSWER SECTION: browser.cloud.raytrek.com. 14113 IN     CNAME   browser-10.raytrek.com. browser-10.raytrek.com. 14113   IN      CNAME   ec2-54-226-13-244.compute-1.amazonaws.com. ec2-54-226-13-244.compute-1.amazonaws.com. 155620 IN ...

Open access doctoral theses on de novo genome assembly

I am presently writing my doctoral thesis. So far, I have found four open access doctoral thesis on de novo genome assembly. Table 1: Doctoral theses on de novo assembly in the next-generation sequencing era. Links to citeulike entries are provided on dates. Date Person Institution January 2008 Mark Chaisson University of California, San Diego, United States of America September 2009 Daniel Robert Zerbino Darwin College, University of Cambridge, United Kingdom July 2012 Rayan Chikhi École normale supérieure de Cachan - Antenne de Bretagne, France September 2012 Jared Thomas Simpson Queens’ College, University of Cambridge, United Kingdom

The impact of link-time optimization

Recently, I compared running times with and without link-time optimization. Link-time optimization (LTO) makes programs run faster and it must be done at compilation. Both the GNU Compiler Collection (GCC) and Clang (a frontend for LLVM) call this link-time optimization. The Intel compiler can also do that in a similar process called interprocedural optimization (IPO). This process is called Whole Program Optimization (also needs Link-time Code Generation -- LTCG) in Microsoft Visual Studio The dataset I used for these tests is ERS006494 . It contains 186073978 Illumina DNA sequences. The length of these sequences is 75 nucleotides. I used 8 nodes, each with 2 Intel Xeon X5560 processors (8 cores per node) and 24 GiB of memory. Storage was served by a Lustre file system. Each job therefore had 64 ranks. The version of GCC was 4.7.2. The version of Open-MPI was 1.6.3. The version of Ray was 606be2a7a710a226. The version of Ray Platform was d78e7ec5037c9c9e8a0. The Host Communic...

Tracking my statistics in Heart of the Swarm

The new StarCraft II: Heart of the Swarm is really fun to play. According to sc2ranks.com , I am currently ranked #4862 in North America and #12192 on Earth. I hope to get promoted in the Gold league before the season ends. The 2013 Season 2 ends on 2013-05-01. I am currently #8 in my division -- which is Division Hierarch Delta.

Cost of hosting Ray Cloud Browser during 1 month in AWS EC2

Image
I got this question on Twitter: what has your bill totaled since you've started hosting ray viewer? I guess ray viewer is just another name for Ray Cloud Browser ;-P. For one month, I expect the costs to be at least 2.1915 $ (365.25/12*24*0.003) for my t1.micro spot instance and 6.4000 $ (64 * 0.10) for my 64 GiB Amazon EBS Standard volume . I/O requests are hard to predict, and I did run some I/O intensive workloads that were unrelated to Ray Cloud Browser during this period on my t1.micro instance. My resources are located in the availability zone us-east-1a. Here the pricing history for t1.micro spot instances in the us-east-1a availability zone. Figure 1: Pricing history for t1.micro spot instances in the us-east-1a availability zone. Data are from Amazon Web Services API. The vertical axis is a by-the-hour price. Using the Cost Allocation Feature, I generated ( with a home-baked Ruby script ) a pivot table for my operation costs in the cloud during the month...