2013-01-31

Best Amazon EC2 instance type for Ray Cloud Browser on metagenomes and bacterial genomes

Hi,

I am using spot instances on Amazon Elastic Compute Cloud (EC2) to deploy a few installations of Ray Cloud Browser. Initially, I opted for m1.small instances because the Ray Cloud Browser web service that answers a bunch of HTTP GET API calls was not optimized. Namely, the C++ back-end code was memory-mapping a huge file (~16-20 GiB). This huge bloated binary file was the index -- the source of information about the huge graph describing a given biological sample. A recent patch improved the performance by packing information in every available bit in the binary file, reducing the number of blocks by 75%, hereby enhancing performance as well.

Recently, there were a lot of peaks in the m1.small spot pricing, and I figured out that my use case was all about bursts -- discrete HTTP API calls.


I then looked at the pricing history for the last 3 months, and these pesky peaks seem to be a 2013 thing.

The t1.micro spot instance pricing history also has these sophisticated highs.

With all these recent pricing surges, I decided to provision some standard Elastic Block Storage (EBS) volumes so that my stuff stays in the cloud when the spot market price exceeds what I want to pay as a customer.

Before today, I was using ephemeral volumes, which are basically just EBS storage volumes that vanish when the instance is stopped or terminated.
I don't use ephemeral EBS volumes anymore as they are not useful for my user story.

Running one m1.small on-demand instance costs 47.48 $ per month whereas one single 64-GiB EBS standard volume costs 6.40 $ per month (excluding input/output requests). And running one m1.small spot instance costs around 5.11 $ per month whereas running one t1.micro spot instance costs around 2.19 $ per month.

Therefore, running Ray Cloud Browser on one t1.micro spot instance using one 64-GiB EBS volume costs 8.59 $ per month. Basically, it costs nothing at all considering the cost of other things in genomics research, such as sequencing runs on instruments, bioinformatician-hours, developper-hours, and so on.

And the software called Ray Cloud Browser costs nothing too -- I authored it and it's free software distributed under the GNU General Public License version 3.

I hope that people everywhere use Ray Cloud Browser to visualize their DNA samples !

The cloud is really a nice thing, it removes barriers. But you have to use it to gain experience in order to lower your costs.


Metagenome lumps & artifactual mutations

Hey !

A recent arXiv manuscript by the group of Professor C. Titus Brown revealed "sequencing artefacts" in metagenomes. In this work, the authors discovered topological features in de Bruijn graphs that they called "lumps."

In the Microbiome40 sample (confidential stuff I guess ;-)), a collaborator observed these metagenome lumps using Ray Cloud Browser (public demo on Amazon EC2; source code on GitHub).

Here are some nice pictures of this system.






Ray assembly plugins are shielded against these as the algorithms are well-designed.



Another recent paper in Nucleic Acids Research by a group at the Broad Institute carefully documented "artifactual mutations" due to particular events during sample preparation.

Also using Ray Cloud Browser, we observed these topological features in the de Bruijn graph on Illumina MiSeq data, namely the public dataset called "E.Coli-DH10B-2x250".

Here are a nice picture of a bubble -- a branching point in the graph where two high-coverage are competing. One of them (the strong part, in blue) has a high coverage (>200 X) where as the weak part has a lower coverage around 20-30 X.

20-30X can not be obtained with random sequencing errors. (edited the word not).

edit: "Yes, it can. GGCxG error in Illumina data."

This will look like genuine SNPs.

edit: "remember when you sequence bacteria you are sequencing a population"


2013-01-30

Visualizing autonomous components in de Bruijn graphs

In Ray Cloud Browser (demo on Amazon EC2), I just found some interesting bits: autonomous components occurring in free form in a de Bruijn graph. You can see these yourself by going at the location displayed in the figures below.







I am working on these strange things in the graph because those useless parts consume a sizable amount of compute time. So instead of patching this problem downstream (i.e.: removing the dust), I decided to write Ray Cloud Browser to fix things in the right places (i.e.: removing the source of dust instead of the dust being produced).

2013-01-27

My roadmap for 2013


Table 1: Roadmap for year 2013.






2013-02-15
2013-06-04 in the future (waiting for co-author comments)
Manuscript (Godzaridis et al.) submission to Big Data
2013-07-15
Send my thesis to director and codirector for review.
2013-08-15
Initial thesis submission
2013-08-31
End of Banting & Best doctoral scholarship (director will pay me up to Dec. 2013)
2013-10-01
Canadian Institutes of Health Research Fellowship application ( 2012-2013)
2013-10-03
Louis-Berlinguet postdoctoral research scholarship application (2012-2013)
2013-11-01
Banting Postdoctoral Fellowship application (2012-2013)
2013-11-15
Doctoral oral defence
2013-12-15
Final thesis submission
2014-04-15
Starting post-doc with Rick Stevens





Needs Ray Cloud Browser paper in there also but the super enhanced visualization zombie version

Edits:

2013-01-27 moved defence from 2013-09-15 to 2013-11-15
2013-01-27 added zombie version

2013-01-25

Measuring latency in the cloud

Hey,

I started 10 cc2.8xlarge in Amazon EC2 using the spot instance market.

The cc2.8xlarge hourly rate sits at 0.270 $ / h. The on-demand rate is 2.400 $ / h.

Results

Table 1: Cloud latencies. Number of instances was 10 and the instance type was cc2.8xlarge.

MPI ranks
MPI ranks per instance
Average roundtrip latency (microseconds)
10
1
235.200785
20
2
313.960899
40
4
365.403384
80
8
473.863469
120
12
563.779653
160
16
322.942884
200
20
258.164757
250
25
220.151894
320
32
280.532563


Instances


The specification of 1 cc2.8xlarge is:

Cluster Compute Eight Extra Large Instance

60.5 GiB of memory
88 EC2 Compute Units (2 x Intel Xeon E5-2670, eight-core)
3370 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
EBS-Optimized Available: No*
API name: cc2.8xlarge


(that's 32 VCPUs with hyperthreading !)

These are the 10 instances in my placement group:

i-b078b9c0: ec2-174-129-96-90.compute-1.amazonaws.com
i-ba78b9ca: ec2-54-234-6-223.compute-1.amazonaws.com
i-b278b9c2: ec2-107-21-145-1.compute-1.amazonaws.com
i-b678b9c6: ec2-204-236-254-68.compute-1.amazonaws.com
i-b878b9c8: ec2-50-16-171-238.compute-1.amazonaws.com
i-bc78b9cc: ec2-54-234-75-41.compute-1.amazonaws.com
i-a678b9d6: ec2-23-22-107-180.compute-1.amazonaws.com
i-a878b9d8: ec2-23-22-77-54.compute-1.amazonaws.com
i-aa78b9da: ec2-23-21-6-134.compute-1.amazonaws.com
i-9678b9e6: ec2-50-16-135-211.compute-1.amazonaws.com

Software used to measuring latency


I will test the latency using latency_checker.

Latency checker uses a any-to-any communication pattern and the default message size is 4000 bytes (excluding envelope).

First, we have to setup keys.

I then tested the 10 links:

[ec2-user@ip-10-156-160-75 ~]$ cat hosts.txt
ec2-174-129-96-90.compute-1.amazonaws.com
ec2-54-234-6-223.compute-1.amazonaws.com
ec2-107-21-145-1.compute-1.amazonaws.com
ec2-204-236-254-68.compute-1.amazonaws.com
ec2-50-16-171-238.compute-1.amazonaws.com
ec2-54-234-75-41.compute-1.amazonaws.com
ec2-23-22-107-180.compute-1.amazonaws.com
ec2-23-22-77-54.compute-1.amazonaws.com
ec2-23-21-6-134.compute-1.amazonaws.com
ec2-50-16-135-211.compute-1.amazonaws.com

[ec2-user@ip-10-156-160-75 ~]$  for i in $(cat hosts.txt ); do ssh $i hostname; done
ip-10-156-160-75
ip-10-156-210-34
ip-10-156-164-15
ip-10-156-210-142
ip-10-156-209-237
ip-10-156-210-243
ip-10-156-160-214
ip-10-156-161-58
ip-10-156-224-53
ip-10-156-225-18

That's great, let's install packages on each instance:

[ec2-user@ip-10-156-160-75 ~]$ sudo yum install -y git openmpi openmpi-devel gcc make; exit

[ec2-user@ip-10-156-160-75 ~]$ mkdir software; cd software
[ec2-user@ip-10-156-160-75 ~]$ git clone git://github.com/sebhtml/latency_checker.git
Cloning into latency_checker...
remote: Counting objects: 51, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 51 (delta 27), reused 49 (delta 25)
Receiving objects: 100% (51/51), 25.14 KiB, done.
Resolving deltas: 100% (27/27), done.

[ec2-user@ip-10-156-160-75 ~]$ cd latency_checker/
[ec2-user@ip-10-156-160-75 latency_checker]$ load_openmpi
[ec2-user@ip-10-156-160-75 latency_checker]$ make
mpicc -O3 -Wall -ansi -DASSERT -c main.c -o main.o
mpicc -O3 -Wall -ansi -DASSERT -c process.c -o process.o
mpicc -O3 -Wall -ansi -DASSERT main.o process.o -o latency_checker

The last step before the actual tests is to copy the executable on all 10 instances.

[ec2-user@ip-10-156-160-75 ~]$ for i in $(cat hosts.txt ); do scp software/latency_checker/latency_checker $i:; done


Measuring latency


[ec2-user@ip-10-156-160-75 ~]$ /usr/lib64/openmpi/bin/mpiexec -hostfile hosts.txt -n 10 ./latency_checker -exchanges 10000 -message-size 4000|grep "average of average roundtrip latencies"
Rank 0 -> average of average roundtrip latencies: 235.200785 microseconds

Software versions:

latency_checker v1.0.0
gcc (GCC) 4.6.2 20111027 (Red Hat 4.6.2-2)
openmpi 1.5.4 mockbuild@gobi-build-31004.sea31.amazon.com
Linux 3.2.30-49.59.amzn1.x86_64

2013-01-21

Assembly of a human genome with latest sequencing technologies

So Illumina released a nice dataset of a human genome with their latest improvements in their sequencing products.

The dataset HiSeq-2500-NA12878-demo-2x150 has 1171357300 reads, each with 151 nucleotides.

Using Ray with the polytope virtual network in Ray Platform, I assembled a pretty good assembly in a timely way. The job ran on 64 nodes or 512 cores, using 512 MPI ranks.

$ cat Ray-polytope.sh
#!/bin/bash

#PBS -S /bin/bash
#PBS -N HiSeq-2500-NA12878-demo-2x150-2013-01-18-2
#PBS -o HiSeq-2500-NA12878-demo-2x150-2013-01-18-2.stdout
#PBS -e HiSeq-2500-NA12878-demo-2x150-2013-01-18-2.stderr
#PBS -A nne-790-ac
#PBS -l walltime=48:00:00
#PBS -l qos=SPJ1024
#PBS -l nodes=64:ppn=8
#PBS -M sebastien.boisvert.3@ulaval.ca
#PBS -m bea
cd $PBS_O_WORKDIR

source /rap/nne-790-ab/software/NGS-Pipelines/LoadModules.sh

mpiexec -n 512 \
-bind-to-core -bynode \
Ray \
-route-messages -connection-type polytope -routing-graph-degree 21 \
 -o \
 HiSeq-2500-NA12878-demo-2x150-2013-01-18-2 \
 -k \
 31 \
-p \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L001_R1_001.fastq.gz \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L001_R2_001.fastq.gz \
-p \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L001_R1_002.fastq.gz \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L001_R2_002.fastq.gz \
-p \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L002_R1_001.fastq.gz \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L002_R2_001.fastq.gz \
-p \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L002_R1_002.fastq.gz \
      HiSeq-2500-NA12878-demo-2x150/sorted_S1_L002_R2_002.fastq.gz \


With the recent optimizations, the slowest step is no longer graph traversal.


tail HiSeq-2500-NA12878-demo-2x150-2013-01-18-2/ElapsedTime.txt -n 24

 Network testing: 1 seconds
 Counting sequences to assemble: 8 minutes, 27 seconds
 Sequence loading: 1 hours, 9 minutes, 12 seconds
 K-mer counting: 31 minutes, 35 seconds
 Coverage distribution analysis: 16 seconds
 Graph construction: 47 minutes, 10 seconds
 Null edge purging: 7 minutes, 21 seconds
 Selection of optimal read markers: 56 minutes, 4 seconds
 Detection of assembly seeds: 3 hours, 11 minutes, 0 seconds
 Estimation of outer distances for paired reads: 16 minutes, 47 seconds
 Bidirectional extension of seeds: 4 hours, 37 minutes, 53 seconds
 Merging of redundant paths: 1 days, 1 hours, 10 minutes, 46 seconds
 Generation of contigs: 4 minutes, 54 seconds
 Scaffolding of contigs: 6 hours, 54 minutes, 46 seconds
 Counting sequences to search: 0 seconds
 Graph coloring: 16 seconds
 Counting contig biological abundances: 4 minutes, 15 seconds
 Counting sequence biological abundances: 0 seconds
 Loading taxons: 13 seconds
 Loading tree: 16 seconds
 Processing gene ontologies: 27 seconds
 Computing neighbourhoods: 0 seconds
 Total: 1 days, 20 hours, 1 minutes, 55 seconds


The assembly is quite good too:

$ cat HiSeq-2500-NA12878-demo-2x150-2013-01-18-2/OutputNumbers.txt
Contigs >= 100 nt
 Number: 632240
 Total length: 2715204138
 Average: 4294
 N50: 9119
 Median: 2234
 Largest: 160698
Contigs >= 500 nt
 Number: 504237
 Total length: 2693905405
 Average: 5342
 N50: 9204
 Median: 3185
 Largest: 160698
Scaffolds >= 100 nt
 Number: 592711
 Total length: 2724259174
 Average: 4596
 N50: 10391
 Median: 2212
 Largest: 160698
Scaffolds >= 500 nt
 Number: 464708
 Total length: 2702960441
 Average: 5816
 N50: 10491
 Median: 3323
 Largest: 160698


Inserts are not that long.

$ grep -A1 AverageO HiSeq-2500-NA12878-demo-2x150-2013-01-18-2/LibraryStatistics.txt
  AverageOuterDistance: 296
  StandardDeviation: 61
--
  AverageOuterDistance: 296
  StandardDeviation: 61
--
  AverageOuterDistance: 296
  StandardDeviation: 61
--
  AverageOuterDistance: 296
  StandardDeviation: 61

2013-01-18

1 human genome, 1 Amazon EC2 instance, x days


Today we are assembling trying to assemble 1 human genome on 1 Amazon EC2 instance using SSD disks. We pay by the hour, 3.1 $ per hour to be exact.
 

Colors


black -> just text explaining stuff
purple -> commands to type on your laptop as a normal user with ec2-api-tools configured (I use a Lenovo X230 with Fedora)
red -> commands to type on the instance as root
blue -> commands to type on the instance as ec2-user

High I/O Quadruple Extra Large Instance

 
60.5 GiB of memory
35 EC2 Compute Units (16 virtual cores*)
2 SSD-based volumes each with 1024 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
Storage I/O Performance: Very High**
EBS-Optimized Available: No***
API name: hi1.4xlarge

Blog post

Pricing (on-demand for Linux/UNIX usage)


$3.100 per Hour


Starting the instance


1..2..3.. let's start!

$ ec2-authorize default -p 22

$ ec2-run-instances ami-08249861 -t hi1.4xlarge -n 1 -b /dev/sdf=ephemeral0 -b /dev/sdg=ephemeral1 -k GoldThor -g default

$ ec2-describe-instances |grep INS|grep hi1.4xlarge|grep -v terminated
INSTANCE    i-a7e6d4d6    ami-08249861    ec2-23-22-97-73.compute-1.amazonaws.com    ip-10-148-132-199.ec2.internal    running    GoldThor    0    hi1.4xlarge    2013-01-18T18:54:44+0000    us-east-1c                monitoring-disabled    23.22.97.73    10.148.132.199        ebs                    hvm    xen        sg-6aa44c02    default    false   

Connecting to the cloud


$ ssh -i GoldThor.pem  ec2-user@ec2-23-22-97-73.compute-1.amazonaws.com


Using SSD disks for virtual memory


[root@ip-10-148-132-199 ec2-user]# mkswap /dev/xvdf  -f
Setting up swapspace version 1, size = 1073741820 KiB
no label, UUID=b1cc9a0e-3fdb-47b3-bdde-3d0fdfadcf20

[root@ip-10-148-132-199 ec2-user]# mkswap /dev/xvdg  -f
Setting up swapspace version 1, size = 1073741820 KiB
no label, UUID=f452b610-80ea-4b5d-b3a9-751ba1e32f55

[root@ip-10-148-132-199 ec2-user]# swapon /dev/xvdf

[root@ip-10-148-132-199 ec2-user]# swapon /dev/xvdg

[ec2-user@ip-10-148-132-199 ~]$ grep Total /proc/meminfo |head -n2
MemTotal:       61370008 kB
SwapTotal:      2147483640 kB

Finding a cool dataset


$ ec2-describe-snapshots -a |grep Illumina|grep Jay|grep Linux
SNAPSHOT    snap-53b3cb3a    vol-ce0dcba7    completed    2010-01-10T20:48:42+0000    100%    947081328633    350    Illumina - Jay Flatley Genome Data Set (Linux)

Here, I used provisioned, but it does not matter because the snapshot takes a while anyway to transit from Amazon S3 to Amazon EBS.

$ ec2-create-volume --snapshot  snap-53b3cb3a -z  us-east-1c -t io1  --iops 2000
VOLUME    vol-c2d096bc    350    snap-53b3cb3a    us-east-1c    creating    2013-01-18T21:01:07+0000    io1    2000

$ ec2-attach-volume vol-c2d096bc -i   i-a7e6d4d6   -d /dev/sdx
ATTACHMENT    vol-c2d096bc    i-a7e6d4d6    /dev/sdx    attaching    2013-01-18T21:02:25+0000   

[root@ip-10-148-132-199 ec2-user]# mkdir /mnt/dataset
[root@ip-10-148-132-199 ec2-user]# mount /dev/sdx /mnt/dataset/

[root@ip-10-148-132-199 ~]# chown ec2-user /mnt/dataset/Alignments -R


Now we need to touch all the blocks otherwise Ray will just be waiting for transit from S3 to EBS because EBS is copy-on-write, if you don't use a block, then it will eventually be there, but it can take a while because it out-of-order (or should I say it's the order the file system ext3 decides to touch blocks).


[root@ip-10-148-132-199 ~]# dd if=/dev/sdx of=/dev/null bs=10M

(dd took around 1 hour, 52 minutes, rate was 42.7 MiB/s)

It would be more cost-effective to do the dd command from a cheaper instance type.

Installing analysis software


[root@ip-10-148-132-199 ec2-user]# yum install git gcc-c++ make openmpi-devel openmpi


[ec2-user@ip-10-148-132-199 ~]$ git clone git://github.com/sebhtml/ray.git
Cloning into ray...
remote: Counting objects: 21530, done.
remote: Compressing objects: 100% (4255/4255), done.
remote: Total 21530 (delta 17534), reused 21153 (delta 17179)
Receiving objects: 100% (21530/21530), 21.47 MiB | 13.59 MiB/s, done.
Resolving deltas: 100% (17534/17534), done.

[ec2-user@ip-10-148-132-199 ~]$ git clone git://github.com/sebhtml/RayPlatform.git
Cloning into RayPlatform...
remote: Counting objects: 17903, done.
remote: Compressing objects: 100% (3325/3325), done.
remote: Total 17903 (delta 14535), reused 17872 (delta 14504)
Receiving objects: 100% (17903/17903), 20.75 MiB | 4.88 MiB/s, done.
Resolving deltas: 100% (14535/14535), done.

[ec2-user@ip-10-148-132-199 ~]$ cd ray

[ec2-user@ip-10-148-132-199 ray]$ load_openmpi

[ec2-user@ip-10-148-132-199 ray]$ make -j 16 PREFIX=/home/ec2-user/software/ray/v-e2c32ffdbe5963b ASSERT=y CXXFLAGS="-Wall -std=c++98 -O3 -march=native"

[ec2-user@ip-10-148-132-199 ray]$ make install

Installing Ray to /home/ec2-user/software/ray/v-e2c32ffdbe5963b

[ec2-user@ip-10-148-132-199 ray]$ mpiexec  -n 1 /home/ec2-user/software/ray/v-e2c32ffdbe5963b/Ray -version
Ray version 2.1.1-devel
License for Ray: GNU General Public License version 3
RayPlatform version: 1.2.0-devel
License for RayPlatform: GNU Lesser General Public License version 3

MAXKMERLENGTH: 32
Maximum coverage depth stored by CoverageDepth: 4294967295
MAXIMUM_MESSAGE_SIZE_IN_BYTES: 4000 bytes
FORCE_PACKING = n
ASSERT = y
HAVE_LIBZ = n
HAVE_LIBBZ2 = n
PROFILER_COLLECT = n
CLOCK_GETTIME = n
__linux__ = y
_MSC_VER = n
__GNUC__ = y
RAY_32_BITS = n
RAY_64_BITS = y
MPI standard version: MPI 2.1
MPI library: Open-MPI 1.5.4
Compiler: GNU gcc/g++ 4.6.2 20111027 (Red Hat 4.6.2-2)
With SSE 4.2
With hardware pop count

Preparing the job file


[ec2-user@ip-10-148-132-199 ~]$ cat create-job.sh
for i in $(find /mnt/dataset/|grep export.txt$|grep _1.export)
do
        echo "-p \\"
        echo "       $i \\"
        echo "       $(echo $i|sed 's/_1_export/_2_export/g') \\"
done

[ec2-user@ip-10-148-132-199 ~]$ bash create-job.sh  > Run-Ray-on-SSDs.sh


Edit the file to get this:

[ec2-user@ip-10-148-132-199 ~]$ head Run-Ray-on-SSDs.sh
#!/bin/bash

. /etc/profile.d/openmpi.sh
load_openmpi
PATH=/home/ec2-user/software/ray/v-e2c32ffdbe5963b/:$PATH

mpiexec -n 16 Ray -k 31 -o Human-Genome-EC2-2013-01-18-1 \
-p \
       /mnt/dataset/Alignments/090416_EAS013R_0019_FC3159L/s_6_1_export.txt \
       /mnt/dataset/Alignments/090416_EAS013R_0019_FC3159L/s_6_2_export.txt \

 [ec2-user@ip-10-148-132-199 ~]$ chmod +x Run-Ray-on-SSDs.sh


Launching the job


[ec2-user@ip-10-148-132-199 ~]$ ./Run-Ray-on-SSDs.sh  &>  Human-Genome-EC2-2013-01-18-1.log


Update:

Basically, Ray loaded sequences, started to build the graph. But the 16 processes became slow when it started to swap heavily.

It turns out that context switches for swapping memory pages is killing performance after all.

[ec2-user@ip-10-148-132-199 ~]$ tail /proc/28585/status -n2
voluntary_ctxt_switches:        6625309
nonvoluntary_ctxt_switches:     10787012

Regardless, this was a fun experience.

[ec2-user@ip-10-148-132-199 ~]$ top -n 1
Tasks: 142 total,  15 running, 127 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.4%us,  8.7%sy,  0.0%ni, 71.2%id,  8.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  61370008k total, 61057092k used,   312916k free,     3080k buffers
Swap: 2147483640k total, 121548288k used, 2025935352k free,    13840k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                  
28584 ec2-user  20   0 11.0g 3.5g 2224 R 100.0  5.9 219:28.68 Ray                                                                                      
  468 root      20   0     0    0    0 S 88.8  0.0 119:29.33 kswapd0                                                                                   
28583 ec2-user  20   0 11.0g 3.5g 2228 R 86.8  5.9 225:19.30 Ray                                                                                       
28575 ec2-user  20   0 11.0g 3.5g 2228 R 84.8  5.9 208:18.21 Ray                                                                                       
28581 ec2-user  20   0 10.7g 3.5g 2224 R 78.9  5.9 213:03.02 Ray                                                                                       
28586 ec2-user  20   0 11.0g 3.5g 2216 R 78.9  5.9 260:32.81 Ray                                                                                       
28580 ec2-user  20   0 10.8g 3.5g 2224 R 77.0  5.9 216:27.89 Ray                                                                                       
28577 ec2-user  20   0 11.0g 3.5g 2224 R 69.1  5.9 208:44.33 Ray                                                                                       
28571 ec2-user  20   0 11.0g 3.5g 2240 R 63.1  5.9 269:15.16 Ray                                                                                       
28573 ec2-user  20   0 10.9g 3.5g 2228 R 63.1  5.9 207:47.91 Ray                                                                                       
28576 ec2-user  20   0 11.0g 3.5g 2228 D 59.2  5.9 208:28.14 Ray                                                                                       
28579 ec2-user  20   0 11.0g 3.5g 2224 R 53.3  5.9 216:33.98 Ray                                                                                       
28582 ec2-user  20   0 11.0g 3.5g 2224 R 49.3  5.9 214:08.51 Ray                                                                                       
28572 ec2-user  20   0 10.9g 3.5g 2228 R 37.5  5.9 285:00.21 Ray                                                                                       
28578 ec2-user  20   0 11.0g 3.5g 2224 R 27.6  5.9 210:39.44 Ray                                                                                       
28574 ec2-user  20   0 11.0g 3.5g 2228 R 25.7  5.9 206:08.48 Ray                                                                                       
28585 ec2-user  20   0 11.0g 3.5g 2224 D 19.7  5.9 229:11.74 Ray                                                                                       
15519 ec2-user  20   0 15068 1204  888 R  2.0  0.0   0:00.01 top                                                                                       
    1 root      20   0 19400  200  200 S  0.0  0.0   0:02.20 init                                                                                      
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                  
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.90 ksoftirqd/0                                                                               
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0                                                                               
    5 root      20   0     0    0    0 S  0.0  0.0   0:00.05 kworker/u:0                                                                               
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                               
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                                                               
    8 root      20   0     0    0    0 S  0.0  0.0   0:01.44 kworker/1:0                                                                               
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.24 ksoftirqd/1                                                                               
   10 root      20   0     0    0    0 S  0.0  0.0   0:06.15 kworker/0:1   
There was an error in this gadget