Ray and large datasets

Ray is a de novo de Bruijn genome assembler that uses message passing interface. Sébastien Boisvert reports the memory utilization for Illumina CEO Jay T. Flatley (SRA010766) with k=19.


The short read archive hosts several large sequencing datasets. Among these, there are  SRA000271 and SRA010766 -- the two are reads from human genomes: the first is an African male individual (HapMap: NA18507) and the other is Illumina CEO Jay T. Flatley.

Variations in the genome of these individuals can be harnessed by mapping the reads onto a sequence reference -- a set of files containing the genetic blueprint as determined by the investigators of the human genome project.

But when a reference is not available, de novo assembly of the sequencing reads into contiguous sequences is one of the first step toward  acquiring an high-quality reference sequence.

Unparalleled parallelism

In parallel computation, powerful processors can be combined to form an alliance stronger and faster at analyzing problems. One way to construct such computation facilities is to interconnect many computers, and use a paradigm known as message passing interface wherein messages are sent between compute nodes to interleave computation and communication.

To the best of my knowledge, de novo assemblers that use the message passing interface are few: ABySS and Ray. ABySS is a set of programs glued together with make files whereas everything is embedded in one program in Ray.

This makes Ray straightforward to launch on super computers or in the cloud -- using schedulers such as Sun Grid Engine. Ray is compliant with the standard MPI 2.2 and is free/open source software.

SRA010766: Illumina CEO data

There are 164 sequence files in this dataset. Paired reads (note the suffixes _1 and _2) in this dataset have a length of 75.





Number of sequences in the dataset SRA010766


File number File name Number of sequences Sequence length Number of nucleotides
1 SRR033552_1.fastq 40966220 75 3072466500
2 SRR033552_2.fastq 40966220 75 3072466500
3 SRR033553_1.fastq 43599280 75 3269946000
4 SRR033553_2.fastq 43599280 75 3269946000
5 SRR033554_1.fastq 41531088 75 3114831600
6 SRR033554_2.fastq 41531088 75 3114831600
7 SRR033555_1.fastq 42828160 75 3212112000
8 SRR033555_2.fastq 42828160 75 3212112000
9 SRR033556_1.fastq 46543048 75 3490728600
10 SRR033556_2.fastq 46543048 75 3490728600
11 SRR033557_1.fastq 46074808 75 3455610600
12 SRR033557_2.fastq 46074808 75 3455610600
13 SRR033558_1.fastq 42524412 75 3189330900
14 SRR033558_2.fastq 42524412 75 3189330900
15 SRR033559_1.fastq 43244880 75 3243366000
16 SRR033559_2.fastq 43244880 75 3243366000
17 SRR033560_1.fastq 44682732 75 3351204900
18 SRR033560_2.fastq 44682732 75 3351204900
19 SRR033561_1.fastq 35273964 75 2645547300
20 SRR033561_2.fastq 35273964 75 2645547300
21 SRR033562_1.fastq 34751780 75 2606383500
22 SRR033562_2.fastq 34751780 75 2606383500
23 SRR033563_1.fastq 35911452 75 2693358900
24 SRR033563_2.fastq 35911452 75 2693358900
25 SRR033564_1.fastq 34959916 75 2621993700
26 SRR033564_2.fastq 34959916 75 2621993700
27 SRR033565_1.fastq 33651596 75 2523869700
28 SRR033565_2.fastq 33651596 75 2523869700
29 SRR033566_1.fastq 27935216 75 2095141200
30 SRR033566_2.fastq 27935216 75 2095141200
31 SRR033567_1.fastq 23905716 75 1792928700
32 SRR033567_2.fastq 23905716 75 1792928700
33 SRR033568_1.fastq 21055424 75 1579156800
34 SRR033568_2.fastq 21055424 75 1579156800
35 SRR033569_1.fastq 54777660 75 4108324500
36 SRR033569_2.fastq 54777660 75 4108324500
37 SRR033570_1.fastq 53258444 75 3994383300
38 SRR033570_2.fastq 53258444 75 3994383300
39 SRR033571_1.fastq 54960632 75 4122047400
40 SRR033571_2.fastq 54960632 75 4122047400
41 SRR033572_1.fastq 52316296 75 3923722200
42 SRR033572_2.fastq 52316296 75 3923722200
43 SRR033573_1.fastq 39423228 75 2956742100
44 SRR033573_2.fastq 39423228 75 2956742100
45 SRR033574_1.fastq 38145116 75 2860883700
46 SRR033574_2.fastq 38145116 75 2860883700
47 SRR033575_1.fastq 43014116 75 3226058700
48 SRR033575_2.fastq 43014116 75 3226058700
49 SRR033576_1.fastq 43881552 75 3291116400
50 SRR033576_2.fastq 43881552 75 3291116400
51 SRR033577_1.fastq 44275780 75 3320683500
52 SRR033577_2.fastq 44275780 75 3320683500
53 SRR033578_1.fastq 50932256 75 3819919200
54 SRR033578_2.fastq 50932256 75 3819919200
55 SRR033579_1.fastq 50963792 75 3822284400
56 SRR033579_2.fastq 50963792 75 3822284400
57 SRR033580_1.fastq 47329708 75 3549728100
58 SRR033580_2.fastq 47329708 75 3549728100
59 SRR033581_1.fastq 36354620 75 2726596500
60 SRR033581_2.fastq 36354620 75 2726596500
61 SRR033582_1.fastq 42518488 75 3188886600
62 SRR033582_2.fastq 42518488 75 3188886600
63 SRR033583_1.fastq 40551820 75 3041386500
64 SRR033583_2.fastq 40551820 75 3041386500
65 SRR033584_1.fastq 41197920 75 3089844000
66 SRR033584_2.fastq 41197920 75 3089844000
67 SRR033585_1.fastq 38647136 75 2898535200
68 SRR033585_2.fastq 38647136 75 2898535200
69 SRR033586_1.fastq 37941964 75 2845647300
70 SRR033586_2.fastq 37941964 75 2845647300
71 SRR033587_1.fastq 45184196 75 3388814700
72 SRR033587_2.fastq 45184196 75 3388814700
73 SRR033588_1.fastq 49829552 75 3737216400
74 SRR033588_2.fastq 49829552 75 3737216400
75 SRR033589_1.fastq 49263392 75 3694754400
76 SRR033589_2.fastq 49263392 75 3694754400
77 SRR033590_1.fastq 46582328 75 3493674600
78 SRR033590_2.fastq 46582328 75 3493674600
79 SRR033591_1.fastq 49959800 75 3746985000
80 SRR033591_2.fastq 49959800 75 3746985000
81 SRR033592_1.fastq 49806624 75 3735496800
82 SRR033592_2.fastq 49806624 75 3735496800
83 SRR033593_1.fastq 44973264 75 3372994800
84 SRR033593_2.fastq 44973264 75 3372994800
85 SRR033594_1.fastq 35298140 75 2647360500
86 SRR033594_2.fastq 35298140 75 2647360500
87 SRR033595_1.fastq 33396224 75 2504716800
88 SRR033595_2.fastq 33396224 75 2504716800
89 SRR033596_1.fastq 33513072 75 2513480400
90 SRR033596_2.fastq 33513072 75 2513480400
91 SRR033597_1.fastq 33738536 75 2530390200
92 SRR033597_2.fastq 33738536 75 2530390200
93 SRR033598_1.fastq 33698860 75 2527414500
94 SRR033598_2.fastq 33698860 75 2527414500
95 SRR033599_1.fastq 34741596 75 2605619700
96 SRR033599_2.fastq 34741596 75 2605619700
97 SRR033600_1.fastq 35548768 75 2666157600
98 SRR033600_2.fastq 35548768 75 2666157600
99 SRR033601_1.fastq 30653280 75 2298996000
100 SRR033601_2.fastq 30653280 75 2298996000
101 SRR033602_1.fastq 31295988 75 2347199100
102 SRR033602_2.fastq 31295988 75 2347199100
103 SRR033603_1.fastq 22230280 75 1667271000
104 SRR033603_2.fastq 22230280 75 1667271000
105 SRR033604_1.fastq 22552392 75 1691429400
106 SRR033604_2.fastq 22552392 75 1691429400
107 SRR033605_1.fastq 22204860 75 1665364500
108 SRR033605_2.fastq 22204860 75 1665364500
109 SRR033606_1.fastq 24272588 75 1820444100
110 SRR033606_2.fastq 24272588 75 1820444100
111 SRR033607_1.fastq 25241752 75 1893131400
112 SRR033607_2.fastq 25241752 75 1893131400
113 SRR033608_1.fastq 25934688 75 1945101600
114 SRR033608_2.fastq 25934688 75 1945101600
115 SRR033609_1.fastq 29209040 75 2190678000
116 SRR033609_2.fastq 29209040 75 2190678000
117 SRR033610_1.fastq 30452744 75 2283955800
118 SRR033610_2.fastq 30452744 75 2283955800
119 SRR033611_1.fastq 33816544 75 2536240800
120 SRR033611_2.fastq 33816544 75 2536240800
121 SRR033612_1.fastq 34501164 75 2587587300
122 SRR033612_2.fastq 34501164 75 2587587300
123 SRR033613_1.fastq 29660620 75 2224546500
124 SRR033613_2.fastq 29660620 75 2224546500
125 SRR033614_1.fastq 37914656 75 2843599200
126 SRR033614_2.fastq 37914656 75 2843599200
127 SRR033615_1.fastq 38005976 75 2850448200
128 SRR033615_2.fastq 38005976 75 2850448200
129 SRR033616_1.fastq 43965744 75 3297430800
130 SRR033616_2.fastq 43965744 75 3297430800
131 SRR033617_1.fastq 46335384 75 3475153800
132 SRR033617_2.fastq 46335384 75 3475153800
133 SRR033618_1.fastq 46948232 75 3521117400
134 SRR033618_2.fastq 46948232 75 3521117400
135 SRR033619_1.fastq 47209944 75 3540745800
136 SRR033619_2.fastq 47209944 75 3540745800
137 SRR033620_1.fastq 45119616 75 3383971200
138 SRR033620_2.fastq 45119616 75 3383971200
139 SRR033621_1.fastq 26939464 75 2020459800
140 SRR033621_2.fastq 26939464 75 2020459800
141 SRR033622_1.fastq 25720336 75 1929025200
142 SRR033622_2.fastq 25720336 75 1929025200
143 SRR033623_1.fastq 27676576 75 2075743200
144 SRR033623_2.fastq 27676576 75 2075743200
145 SRR033624_1.fastq 31982932 75 2398719900
146 SRR033624_2.fastq 31982932 75 2398719900
147 SRR033625_1.fastq 35308472 75 2648135400
148 SRR033625_2.fastq 35308472 75 2648135400
149 SRR033626_1.fastq 35703328 75 2677749600
150 SRR033626_2.fastq 35703328 75 2677749600
151 SRR033627_1.fastq 33266868 75 2495015100
152 SRR033627_2.fastq 33266868 75 2495015100
153 SRR033628_1.fastq 53547956 75 4016096700
154 SRR033628_2.fastq 53547956 75 4016096700
155 SRR033629_1.fastq 45474980 75 3410623500
156 SRR033629_2.fastq 45474980 75 3410623500
157 SRR033630_1.fastq 40327452 75 3024558900
158 SRR033630_2.fastq 40327452 75 3024558900
159 SRR033631_1.fastq 41158436 75 3086882700
160 SRR033631_2.fastq 41158436 75 3086882700
161 SRR033632_1.fastq 54755312 75 4106648400
162 SRR033632_2.fastq 54755312 75 4106648400
163 SRR033633_1.fastq 41212468 75 3090935100
164 SRR033633_2.fastq 41212468 75 3090935100
Total - 6372129288 - 477909696600


Assuming the genome size is 3 gigabases, the coverage is 159.


Memory utilization

To run and develop Ray on this large dataset, I utilize colosse: 960 compute nodes each having two quad-core Intel Nehalem-EP -- 8 cores -- and 24 gigabytes of memory. I runned Ray on 64 computers: 512 processors cores and 1536 gigabytes of distributed memory.

Utilization of resources
 
Global identifierPer-host identifierHost-specific queue-nameTypeJob nameJob identifierHost nameAverage loadNumber of processor coresHost-specific utilized memoryHost-specific available memory
11mpi@r104-n27MASTERRay.r41401184258r104-n278.20820.4G23.5G
11mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
22mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
33mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
44mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
55mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
66mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
77mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
88mpi@r104-n27SLAVERay.r41401184258r104-n278.20820.4G23.5G
91mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
102mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
113mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
124mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
135mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
146mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
157mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
168mpi@r105-n49SLAVERay.r41401184258r105-n498.10820.2G23.5G
171mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
182mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
193mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
204mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
215mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
226mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
237mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
248mpi@r105-n51SLAVERay.r41401184258r105-n518.14820.1G23.5G
251mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
262mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
273mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
284mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
295mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
306mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
317mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
328mpi@r106-n27SLAVERay.r41401184258r106-n278.04820.6G23.5G
331mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
342mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
353mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
364mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
375mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
386mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
397mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
408mpi@r106-n29SLAVERay.r41401184258r106-n298.13820.1G23.5G
411mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
422mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
433mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
444mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
455mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
466mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
477mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
488mpi@r106-n35SLAVERay.r41401184258r106-n358.00820.6G23.5G
491mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
502mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
513mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
524mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
535mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
546mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
557mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
568mpi@r106-n36SLAVERay.r41401184258r106-n368.00820.8G23.5G
571mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
582mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
593mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
604mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
615mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
626mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
637mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
648mpi@r106-n41SLAVERay.r41401184258r106-n418.00820.2G23.5G
651mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
662mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
673mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
684mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
695mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
706mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
717mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
728mpi@r106-n50SLAVERay.r41401184258r106-n508.20820.0G23.5G
731mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
742mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
753mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
764mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
775mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
786mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
797mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
808mpi@r106-n60SLAVERay.r41401184258r106-n608.00820.4G23.5G
811mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
822mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
833mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
844mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
855mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
866mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
877mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
888mpi@r106-n61SLAVERay.r41401184258r106-n6110.00821.3G23.5G
891mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
902mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
913mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
924mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
935mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
946mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
957mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
968mpi@r106-n69SLAVERay.r41401184258r106-n698.00820.0G23.5G
971mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
982mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
993mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
1004mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
1015mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
1026mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
1037mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
1048mpi@r107-n52SLAVERay.r41401184258r107-n528.03820.1G23.5G
1051mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1062mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1073mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1084mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1095mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1106mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1117mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1128mpi@r107-n73SLAVERay.r41401184258r107-n738.12820.6G23.5G
1131mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1142mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1153mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1164mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1175mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1186mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1197mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1208mpi@r107-n74SLAVERay.r41401184258r107-n748.11820.4G23.5G
1211mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1222mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1233mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1244mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1255mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1266mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1277mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1288mpi@r107-n77SLAVERay.r41401184258r107-n778.12820.6G23.5G
1291mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1302mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1313mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1324mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1335mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1346mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1357mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1368mpi@r107-n78SLAVERay.r41401184258r107-n788.13820.6G23.5G
1371mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1382mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1393mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1404mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1415mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1426mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1437mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1448mpi@r107-n88SLAVERay.r41401184258r107-n888.02820.1G23.5G
1451mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1462mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1473mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1484mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1495mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1506mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1517mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1528mpi@r107-n92SLAVERay.r41401184258r107-n928.15820.4G23.5G
1531mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1542mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1553mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1564mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1575mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1586mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1597mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1608mpi@r108-n35SLAVERay.r41401184258r108-n358.00820.5G23.5G
1611mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1622mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1633mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1644mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1655mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1666mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1677mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1688mpi@r108-n40SLAVERay.r41401184258r108-n408.13820.0G23.5G
1691mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1702mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1713mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1724mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1735mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1746mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1757mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1768mpi@r108-n41SLAVERay.r41401184258r108-n418.13820.1G23.5G
1771mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1782mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1793mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1804mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1815mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1826mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1837mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1848mpi@r108-n47SLAVERay.r41401184258r108-n478.00820.5G23.5G
1851mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1862mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1873mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1884mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1895mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1906mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1917mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1928mpi@r108-n48SLAVERay.r41401184258r108-n488.00820.4G23.5G
1931mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
1942mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
1953mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
1964mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
1975mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
1986mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
1997mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
2008mpi@r108-n59SLAVERay.r41401184258r108-n598.15820.8G23.5G
2011mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2022mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2033mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2044mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2055mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2066mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2077mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2088mpi@r108-n64SLAVERay.r41401184258r108-n648.00820.5G23.5G
2091mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2102mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2113mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2124mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2135mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2146mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2157mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2168mpi@r108-n77SLAVERay.r41401184258r108-n778.00821.0G23.5G
2171mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2182mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2193mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2204mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2215mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2226mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2237mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2248mpi@r108-n84SLAVERay.r41401184258r108-n848.02820.6G23.5G
2251mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2262mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2273mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2284mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2295mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2306mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2317mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2328mpi@r108-n91SLAVERay.r41401184258r108-n918.06820.7G23.5G
2331mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2342mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2353mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2364mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2375mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2386mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2397mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2408mpi@r109-n1SLAVERay.r41401184258r109-n18.14820.4G23.5G
2411mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2422mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2433mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2444mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2455mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2466mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2477mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2488mpi@r109-n10SLAVERay.r41401184258r109-n108.11820.5G23.5G
2491mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2502mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2513mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2524mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2535mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2546mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2557mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2568mpi@r109-n12SLAVERay.r41401184258r109-n128.01820.2G23.5G
2571mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2582mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2593mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2604mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2615mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2626mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2637mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2648mpi@r109-n13SLAVERay.r41401184258r109-n138.13820.6G23.5G
2651mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2662mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2673mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2684mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2695mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2706mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2717mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2728mpi@r109-n14SLAVERay.r41401184258r109-n148.00820.5G23.5G
2731mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2742mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2753mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2764mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2775mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2786mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2797mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2808mpi@r109-n15SLAVERay.r41401184258r109-n158.17820.1G23.5G
2811mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2822mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2833mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2844mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2855mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2866mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2877mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2888mpi@r109-n16SLAVERay.r41401184258r109-n168.01820.5G23.5G
2891mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2902mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2913mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2924mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2935mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2946mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2957mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2968mpi@r109-n18SLAVERay.r41401184258r109-n188.00820.5G23.5G
2971mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
2982mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
2993mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
3004mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
3015mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
3026mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
3037mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
3048mpi@r109-n19SLAVERay.r41401184258r109-n198.14820.6G23.5G
3051mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3062mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3073mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3084mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3095mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3106mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3117mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3128mpi@r109-n21SLAVERay.r41401184258r109-n218.16820.3G23.5G
3131mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3142mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3153mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3164mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3175mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3186mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3197mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3208mpi@r109-n22SLAVERay.r41401184258r109-n228.00820.6G23.5G
3211mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3222mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3233mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3244mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3255mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3266mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3277mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3288mpi@r109-n23SLAVERay.r41401184258r109-n238.13820.5G23.5G
3291mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3302mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3313mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3324mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3335mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3346mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3357mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3368mpi@r109-n3SLAVERay.r41401184258r109-n38.00820.5G23.5G
3371mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3382mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3393mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3404mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3415mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3426mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3437mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3448mpi@r109-n4SLAVERay.r41401184258r109-n48.05820.6G23.5G
3451mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3462mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3473mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3484mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3495mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3506mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3517mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3528mpi@r109-n40SLAVERay.r41401184258r109-n408.06820.4G23.5G
3531mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3542mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3553mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3564mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3575mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3586mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3597mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3608mpi@r109-n43SLAVERay.r41401184258r109-n438.00821.0G23.5G
3611mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3622mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3633mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3644mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3655mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3666mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3677mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3688mpi@r109-n5SLAVERay.r41401184258r109-n58.00820.5G23.5G
3691mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3702mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3713mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3724mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3735mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3746mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3757mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3768mpi@r109-n50SLAVERay.r41401184258r109-n508.14820.2G23.5G
3771mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3782mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3793mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3804mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3815mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3826mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3837mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3848mpi@r109-n52SLAVERay.r41401184258r109-n528.00820.3G23.5G
3851mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3862mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3873mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3884mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3895mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3906mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3917mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3928mpi@r109-n58SLAVERay.r41401184258r109-n588.02820.6G23.5G
3931mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
3942mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
3953mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
3964mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
3975mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
3986mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
3997mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
4008mpi@r109-n60SLAVERay.r41401184258r109-n608.15820.2G23.5G
4011mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4022mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4033mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4044mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4055mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4066mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4077mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4088mpi@r109-n61SLAVERay.r41401184258r109-n618.00820.6G23.5G
4091mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4102mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4113mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4124mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4135mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4146mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4157mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4168mpi@r109-n62SLAVERay.r41401184258r109-n628.00820.2G23.5G
4171mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4182mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4193mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4204mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4215mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4226mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4237mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4248mpi@r109-n63SLAVERay.r41401184258r109-n638.00820.2G23.5G
4251mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4262mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4273mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4284mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4295mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4306mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4317mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4328mpi@r109-n65SLAVERay.r41401184258r109-n658.00820.3G23.5G
4331mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4342mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4353mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4364mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4375mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4386mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4397mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4408mpi@r109-n66SLAVERay.r41401184258r109-n668.00820.2G23.5G
4411mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4422mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4433mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4444mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4455mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4466mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4477mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4488mpi@r109-n7SLAVERay.r41401184258r109-n78.14820.6G23.5G
4491mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4502mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4513mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4524mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4535mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4546mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4557mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4568mpi@r109-n8SLAVERay.r41401184258r109-n88.14820.7G23.5G
4571mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4582mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4593mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4604mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4615mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4626mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4637mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4648mpi@r109-n9SLAVERay.r41401184258r109-n98.14820.6G23.5G
4651mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4662mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4673mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4684mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4695mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4706mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4717mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4728mpi@r110-n11SLAVERay.r41401184258r110-n118.04820.5G23.5G
4731mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4742mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4753mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4764mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4775mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4786mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4797mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4808mpi@r110-n14SLAVERay.r41401184258r110-n148.15820.2G23.5G
4811mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4822mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4833mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4844mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4855mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4866mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4877mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4888mpi@r110-n15SLAVERay.r41401184258r110-n158.00820.1G23.5G
4891mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4902mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4913mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4924mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4935mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4946mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4957mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4968mpi@r110-n19SLAVERay.r41401184258r110-n198.15820.2G23.5G
4971mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
4982mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
4993mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
5004mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
5015mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
5026mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
5037mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
5048mpi@r110-n2SLAVERay.r41401184258r110-n28.15820.2G23.5G
5051mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5062mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5073mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5084mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5095mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5106mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5117mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G
5128mpi@r110-n5SLAVERay.r41401184258r110-n58.00820.1G23.5G

If you have read the table thoroughly, you presumably noticed the load of the compute node named r106-n61. It is 10.00 and it is off-limits. Reason's zombies. Yes, zombies. Previous jobs were not properly cleaned on completion and ghost processes plague some compute nodes, sometimes.



Comments

Unknown said…
Hi,

I'm really interested in this:
- how long did this job take?

We have some people using Velvet, but they require huge memory machines (512 GB+), and we have clusters similar to this, with much less on individual nodes.

Popular posts from this blog

Le tissu adipeux brun, la thermogénèse, et les bains froids

My 2022 Calisthenics split routine

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor