The Bioinformatics Adventure

Ray Platform: a message-passing-interface programming framework

2012-01-17T09:39:00.000-08:00

The first pieces of Ray were put together in a git repository on 21 January 2010. Since that, Ray has undergo numerous major refactoring events.

The biggest changes to date are those of a few days ago. The source code of Ray is now divided in Ray Platform (around 13000 source lines of code including comments and blank lines) and Ray Application (around 29 000 source lines of code including comments and blank lines).

Ray Platform compiles independently of Ray Application -- it can be reused over and over again. I changed the license too. While Ray Application is still licensed under the GNU General Public License, version 3, Ray Platform is licensed under the GNU Lesser General Public License, version 3.

This means that Ray Platform can be incorporated in proprietary software (for free, if that matters). If changes are made to Ray Platform, however, these have to be shared.

How it works ?

The main class of the Ray Platform is ComputeCore. It represents a process mapped to a single processor core. An instance of this class has a message inbox, a message outbox, a virtual processor and a virtual communicator. The programmer can use the application programming interface to register callbacks for message tags, slave modes and master modes.

Message tag handlers

Any MPI message tag (type of message) has a default callback method to handle it that does nothing at all. Let's say the MPI tag is called MPI_TAG_GET_COFFEE. The corresponding callback is call_MPI_TAG_GET_COFFEE. The empty callbacks are generated in MessageTagHandler automatically. To overwrite one, a class that inherits from MessageTagHandler has to be defined with the appropriate new handler (with the same name). Then, the programmer needs to register an instance of this new class as being the handler of the MPI tag.

m_computeCore.setMessageTagObjectHandler(MPI_TAG_GET_COFFEE, &m_coffeeHandler);

Henceforth, when a message is received, Ray Platform will call m_coffeeHandler.call_MPI_TAG_GET_COFFEE(message).

Master and slave modes

In the Ray Platform, each ComputeCore has a current master mode and a current slave mode. The master mode of all ranks but the master is MASTER_RANK_DO_NOTHING. Just like for the message tags, the default handlers for slave modes and master modes can be overwritten with custom ones.

Links

Making a github project readable as a web page

2012-01-07T18:25:00.000-08:00

It is no secret, I love github (and git).

Today I updated my knowledge on javascript.

I started to wrote a game, the slim volleyball game to be exact (see reddit).

So I pushed the thing on github.
However, the files were not served as being part of a web page.

Fortunately, github implements a service called github pages which allows any github user to share a web page about something using the user's username.

This service also allows one to add a branch called 'gh-pages' (refs/heads/gh-pages to be exact I believe) to git repository to serve its content on the web. The content of the branch 'gh-pages' for projectX of userA is served as a web page at http://userA.github.com/projectX.

Initially, the only reference is usually 'master' (along with its remote twin).

seb@godzilla$ git branch -a
* master
  remotes/origin/master

To serve all the content of the branch 'refs/heads/master' on github pages, a symbolic reference called 'refs/heads/gh-pages' must be created.

seb@godzilla$ git symbolic-ref "refs/heads/gh-pages" "refs/heads/master"

Then you can list the branches.

seb@godzilla$ git branch -a
  gh-pages -> master
* master
  remotes/origin/master

The next thing you need to do is to push the references.

seb@godzilla$ git push --mirror

The gh-pages reference is now on the remote site too.

seb@godzilla$ git branch -a
  gh-pages -> master
* master
  remotes/origin/gh-pages
  remotes/origin/master

While the the project is available on github as usual, the same content is served on the web too !

Qui est en charge ?

2011-11-03T21:44:00.000-07:00

Construction d'un bâtiment

2012 commencera bientôt et le Centre de génomique de Québec est toujours en piètre état. Rénovation ici, bruits insupportables là. Et il devait être livré (ou délivré) en 2005. Il faut déjà réparer ses défectuosités majeures.

Selon le Journal de Québec, ce projet aura coûté (au moins) 22 M$.

L'ancien directeur du centre de recherche du CHUQ, Jean-Claude Forest, affirme que l'impact financier est en train d'être calculé.

Mais il est (un peu) tard pour les chercheurs.

Gestion informatique d'étudiants

Un autre échec est le système informatique Capsule de l'Université Laval -- un portail web qui gère (très mal) les informations des étudiants. Ce projet aura coûté 26 M$.

André Armstrong, le directeur du projet de modernisation des études, attribue son échec à la non-participation étudiante à des comités de suivi.

Je trouve difficile à comprendre comment on peut laisser un projet ayant un aussi gros budget échouer autant à cause de sièges étudiants vacants.

Carnet informatisé de santé

"Les médecins pourraient traiter 20 % plus de patients s'ils avaient accès aux dossiers électroniques."

Mais le Dossier (informatique) de santé du Québec, avec un budget de 563 M$, doit maintenant traverser l'étape décisive: l'implémentation d'une interopérabilité. Il est très difficile de modifier des logiciels qui n'ont pas été initialement conçus pour opérer ensemble (inter-opération).

Un autre échec, quoi.

Marché émergent

Les firmes d'avocats adorent. La corruption et la collusion est un marché lucratif pour celles-ci. Il y a même des chercheurs qui poursuivent l'université à la Cours supérieure du Québec (et vice-versa).

Code review: What happens in Open-MPI's MPI_Iprobe ?

2011-09-27T11:01:00.000-07:00

Code review: What happens in Open-MPI's MPI_Iprobe ?

Update 1

Subject: Re: [OMPI devel] Implementation of MPI_Iprobe
From: George Bosilca (bosilca_at_[hidden])
Date: 2011-09-27 15:34:05

Sebastien,

Your analysis is correct in case the checkpoint/restart approach maintained by ORNL is enabled. This is not the code path of the "normal" MPI processes, where the PML OB1 is used. In this generic case the function mca_pml_ob1_iprobe, defined in the file ompi/mca/pml/ob1/pml_ob1_iprobe.c is used.

george.

http://www.open-mpi.org/community/lists/devel/2011/09/9766.php

End of update 1

The message-passing interface (MPI) standard defines an interface for passing messages between processes.

These processes are not necessarily running on the same physical computer.

Open-MPI is an implementation of the MPI standard.

Here, I have utilised openmpi-1.4.3 to find out what is happening when a call to MPI_Iprobe occurs.

According to the MPI 2.2 standard

"The MPI_PROBE and MPI_IPROBE operations allow incoming messages to be checked for, without actually receiving them."

The prototype of MPI_Iprobe is

int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag, MPI_Status *status);

In ompi/mpi/c/iprobe.c, MPI_Iprobe is embodied.

The code (some lines omitted):

ompi/mpi/c/iprobe.c

/*
 * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
 *                         University Research and Technology
 *                         Corporation.  All rights reserved.
 * Copyright (c) 2004-2005 The University of Tennessee and The University
 *                         of Tennessee Research Foundation.  All rights
 *                         reserved.
 * Copyright (c) 2004-2008 High Performance Computing Center Stuttgart, 
 *                         University of Stuttgart.  All rights reserved.
 * Copyright (c) 2004-2005 The Regents of the University of California.
 *                         All rights reserved.
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */

int MPI_Iprobe(int source, int tag, MPI_Comm comm, int *flag, MPI_Status *status)
{
    int rc;

    MEMCHECKER(
        memchecker_comm(comm);
    );

    if ( MPI_PARAM_CHECK ) {
        rc = MPI_SUCCESS;
        OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
        if (((tag < 0) && (tag != MPI_ANY_TAG)) || (tag > mca_pml.pml_max_tag)) {
            rc = MPI_ERR_TAG;
        } else if (ompi_comm_invalid(comm)) {
            rc = MPI_ERR_COMM;
        } else if ((source != MPI_ANY_SOURCE) &&
                   (MPI_PROC_NULL != source) &&
                   ompi_comm_peer_invalid(comm, source)) {
            rc = MPI_ERR_RANK;
        }
        OMPI_ERRHANDLER_CHECK(rc, comm, rc, FUNC_NAME);
    }

    if (MPI_PROC_NULL == source) {
        if (MPI_STATUS_IGNORE != status) {
            *status = ompi_request_empty.req_status;
            /*
             * Per MPI-1, the MPI_ERROR field is not defined for single-completion calls
             */
            MEMCHECKER(
                opal_memchecker_base_mem_undefined(&status->MPI_ERROR, sizeof(int));
            );
        }
        return MPI_SUCCESS;
    }

    OPAL_CR_ENTER_LIBRARY();

    rc = MCA_PML_CALL(iprobe(source, tag, comm, flag, status));

    /*
     * Per MPI-1, the MPI_ERROR field is not defined for single-completion calls
     */
    MEMCHECKER(
        opal_memchecker_base_mem_undefined(&status->MPI_ERROR, sizeof(int));
    );

    OMPI_ERRHANDLER_RETURN(rc, comm, rc, FUNC_NAME);
}

In this function, the following line is the one we want to follow.

rc = MCA_PML_CALL(iprobe(source, tag, comm, flag, status));

The rest is mostly just parameter validation, I guess.

MCA_PML_CALL is probably a macro. Let's search it.

In ompi/mca/pml/pml.h, MCA_PML_CALL is defined. Here is the code (some lines omitted):

ompi/mca/pml/pml.h

/*
 * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
 *                         University Research and Technology
 *                         Corporation.  All rights reserved.
 * Copyright (c) 2004-2005 The University of Tennessee and The University
 *                         of Tennessee Research Foundation.  All rights
 *                         reserved.
 * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, 
 *                         University of Stuttgart.  All rights reserved.
 * Copyright (c) 2004-2006 The Regents of the University of California.
 *                         All rights reserved.
 * Copyright (c) 2006-2007 Los Alamos National Security, LLC.  All rights
 *                         reserved. 
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */

#if MCA_pml_DIRECT_CALL

#include MCA_pml_DIRECT_CALL_HEADER

#define MCA_PML_CALL_EXPANDER(a, b) MCA_PML_CALL_STAMP(a,b)
#define MCA_PML_CALL(a) MCA_PML_CALL_EXPANDER(MCA_pml_DIRECT_CALL_COMPONENT, a)

#else
#define MCA_PML_CALL(a) mca_pml.pml_ ## a
#endif

With MCA_PML_CALL, iprobe becomes mca_pml.pml_iprobe and some additional code is added depending on the value of MCA_pml_DIRECT_CALL.

In ompi/mca/pml/pml.h, mca_pml_base_module_t is defined.

ompi/mca/pml/pml.h

/*
 * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
 *                         University Research and Technology
 *                         Corporation.  All rights reserved.
 * Copyright (c) 2004-2005 The University of Tennessee and The University
 *                         of Tennessee Research Foundation.  All rights
 *                         reserved.
 * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, 
 *                         University of Stuttgart.  All rights reserved.
 * Copyright (c) 2004-2006 The Regents of the University of California.
 *                         All rights reserved.
 * Copyright (c) 2006-2007 Los Alamos National Security, LLC.  All rights
 *                         reserved. 
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */

struct mca_pml_base_module_1_0_0_t {

    /* downcalls from MCA to PML */
    mca_pml_base_module_add_procs_fn_t    pml_add_procs;
    mca_pml_base_module_del_procs_fn_t    pml_del_procs;
    mca_pml_base_module_enable_fn_t       pml_enable;
    mca_pml_base_module_progress_fn_t     pml_progress;

    /* downcalls from MPI to PML */
    mca_pml_base_module_add_comm_fn_t     pml_add_comm;
    mca_pml_base_module_del_comm_fn_t     pml_del_comm;
    mca_pml_base_module_irecv_init_fn_t   pml_irecv_init;
    mca_pml_base_module_irecv_fn_t        pml_irecv;
    mca_pml_base_module_recv_fn_t         pml_recv;
    mca_pml_base_module_isend_init_fn_t   pml_isend_init;
    mca_pml_base_module_isend_fn_t        pml_isend;
    mca_pml_base_module_send_fn_t         pml_send;
    mca_pml_base_module_iprobe_fn_t       pml_iprobe;
    mca_pml_base_module_probe_fn_t        pml_probe;
    mca_pml_base_module_start_fn_t        pml_start;

    /* diagnostics */
    mca_pml_base_module_dump_fn_t         pml_dump;

    /* FT Event */
    mca_pml_base_module_ft_event_fn_t     pml_ft_event;

    /* maximum constant sizes */
    int                                   pml_max_contextid;
    int                                   pml_max_tag;
};
typedef struct mca_pml_base_module_1_0_0_t mca_pml_base_module_1_0_0_t;
typedef mca_pml_base_module_1_0_0_t mca_pml_base_module_t;

In mca_pml_base_module_t, there is the attribute pml_iprobe. This type of this attribute is mca_pml_base_module_iprobe_fn_t.

This type is defined also in ompi/mca/pml/pml.h and is basically a function pointer (right?).

ompi/mca/pml/pml.h

/*
 * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
 *                         University Research and Technology
 *                         Corporation.  All rights reserved.
 * Copyright (c) 2004-2005 The University of Tennessee and The University
 *                         of Tennessee Research Foundation.  All rights
 *                         reserved.
 * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, 
 *                         University of Stuttgart.  All rights reserved.
 * Copyright (c) 2004-2006 The Regents of the University of California.
 *                         All rights reserved.
 * Copyright (c) 2006-2007 Los Alamos National Security, LLC.  All rights
 *                         reserved. 
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */

/**
 * Probe to poll for pending recv.
 *
 * @param src (IN)        Source rank w/in communicator.
 * @param tag (IN)        User defined tag.
 * @param comm (IN)       Communicator.
 * @param matched (OUT)   Flag indicating if matching recv exists.
 * @param status (OUT)    Completion statuses.
 * @return                OMPI_SUCCESS or failure status.
 *
 */
typedef int (*mca_pml_base_module_iprobe_fn_t)(
    int src,
    int tag,
    struct ompi_communicator_t* comm,
    int *matched,
    ompi_status_public_t *status
);

I believe that this scheme allows to have several implementations of pml_iprobe.

Let us look for these now.

In ompi/mca/pml/crcpw/pml_crcpw_module.c, there is this code (some lines omitted):

ompi/mca/pml/crcpw/pml_crcpw_module.c

/*
 * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana
 *                         University Research and Technology
 *                         Corporation.  All rights reserved.
 * Copyright (c) 2004-2006 The University of Tennessee and The University
 *                         of Tennessee Research Foundation.  All rights
 *                         reserved.
 * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart,
 *                         University of Stuttgart.  All rights reserved.
 * Copyright (c) 2004-2006 The Regents of the University of California.
 *                         All rights reserved.
 * $COPYRIGHT$
 *
 * Additional copyrights may follow
 *
 * $HEADER$
 */

int mca_pml_crcpw_iprobe(int dst, int tag, struct ompi_communicator_t* comm, int *matched, ompi_status_public_t* status )
{
    int ret;
    ompi_crcp_base_pml_state_t * pml_state;

    PML_CRCP_STATE_ALLOC(pml_state, ret);

    pml_state->wrapped_pml_component = &(mca_pml_crcpw_module.wrapped_pml_component);
    pml_state->wrapped_pml_module    = &(mca_pml_crcpw_module.wrapped_pml_module);

    pml_state->state = OMPI_CRCP_PML_PRE;
    pml_state = ompi_crcp.pml_iprobe(dst, tag, comm, matched, status, pml_state);
    if( OMPI_SUCCESS != pml_state->error_code) {
        ret =  pml_state->error_code;
        PML_CRCP_STATE_RETURN(pml_state);
        return ret;
    }

    if( OMPI_CRCP_PML_DONE == pml_state->state) {
        goto CLEANUP;
    }

    if( OMPI_CRCP_PML_SKIP != pml_state->state) {
        if( OMPI_SUCCESS != (ret = mca_pml_crcpw_module.wrapped_pml_module.pml_iprobe(dst, tag, comm, matched, status) ) ) {
            PML_CRCP_STATE_RETURN(pml_state);
            return ret;
        }
    }

    pml_state->state = OMPI_CRCP_PML_POST;
    pml_state = ompi_crcp.pml_iprobe(dst, tag, comm, matched, status, pml_state);
    if( OMPI_SUCCESS != pml_state->error_code) {
        ret =  pml_state->error_code;
        PML_CRCP_STATE_RETURN(pml_state);
        return ret;
    }

 CLEANUP:
    PML_CRCP_STATE_RETURN(pml_state);

    return OMPI_SUCCESS;
}

This code however seems to rely heavily on ompi_crcp.pml_iprobe.

In ompi/mca/crcp/bkmrk/crcp_bkmrk_pml.c (some lines omitted):

ompi/mca/crcp/bkmrk/crcp_bkmrk_pml.c

/*
 * Copyright (c) 2004-2008 The Trustees of Indiana University.
 *                         All rights reserved.
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */
/**************** Probe *****************/
/* JJH - Code reuse: Combine iprobe and probe logic */
ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_iprobe(
                                  int dst, int tag,
                                  struct ompi_communicator_t* comm,
                                  int *matched,
                                  ompi_status_public_t* status,
                                  ompi_crcp_base_pml_state_t* pml_state )
{
    ompi_crcp_bkmrk_pml_drain_message_ref_t   *drain_msg_ref = NULL;
    ompi_crcp_bkmrk_pml_message_content_ref_t *content_ref = NULL;
    int exit_status = OMPI_SUCCESS;
    int ret;

    OPAL_OUTPUT_VERBOSE((30, mca_crcp_bkmrk_component.super.output_handle,
                        "crcp:bkmrk: pml_iprobe(%d, %d)", dst, tag));

    /*
     * Before PML Call
     * - Determine if this can be satisfied from the drained list
     * - Otherwise let the PML handle it
     */
    if( OMPI_CRCP_PML_PRE == pml_state->state) {
        /*
         * Check to see if this message is in the drained message list
         */
        if( OMPI_SUCCESS != (ret = drain_message_find_any(PROBE_ANY_COUNT, tag, dst,
                                                          comm, PROBE_ANY_SIZE,
                                                          &drain_msg_ref,
                                                          &content_ref,
                                                          NULL) ) ) {
            ERROR_SHOULD_NEVER_HAPPEN("crcp:bkmrk: pml_iprobe(): Failed trying to find a drained message.");
            exit_status = ret;
            goto DONE;
        }

        /*
         * If the message is a drained message
         *  - Copy of the status structure to pass back to the user
         *  - Mark the 'matched' flag as true
         */
        if( NULL != drain_msg_ref ) {
            OPAL_OUTPUT_VERBOSE((12, mca_crcp_bkmrk_component.super.output_handle,
                                 "crcp:bkmrk: pml_iprobe(): Matched a drained message..."));

            /* Copy the status information */
            if( MPI_STATUS_IGNORE != status ) {
                memcpy(status, &content_ref->status, sizeof(ompi_status_public_t));
            }

            /* Mark as complete */
            *matched = 1;

            /* This will identify to the wrapper that this message is complete */
            pml_state->state = OMPI_CRCP_PML_DONE;
            pml_state->error_code = OMPI_SUCCESS;
            return pml_state;
        }
        /*
         * Otherwise the message is not drained (common case), so let the PML deal with it
         */
        else {
            /* Mark as not complete */
            *matched = 0;
        }
    }

 DONE:
    pml_state->error_code = exit_status;
    return pml_state;
}

How I understand it, ompi_crcp_bkmrk_pml_iprobe first check if any message in the drained list satisfies what the calling code wants with
a call to drain_message_find_any.

If this is not the case, the code "let the PML deal with it", which probably means that that this case would be dealt with somewhere in mca_pml_crcpw_iprobe although I am not sure.

drain_message_find_any is defined in ompi/mca/crcp/bkmrk/crcp_bkmrk_pml.c:

ompi/mca/crcp/bkmrk/crcp_bkmrk_pml.c

/*
 * Copyright (c) 2004-2008 The Trustees of Indiana University.
 *                         All rights reserved.
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */

static int drain_message_find_any(size_t count, int tag, int peer,
                                  struct ompi_communicator_t* comm, size_t ddt_size,
                                  ompi_crcp_bkmrk_pml_drain_message_ref_t ** found_msg_ref,
                                  ompi_crcp_bkmrk_pml_message_content_ref_t ** content_ref,
                                  ompi_crcp_bkmrk_pml_peer_ref_t **peer_ref)
{
    ompi_crcp_bkmrk_pml_peer_ref_t *cur_peer_ref = NULL;
    opal_list_item_t* item = NULL;

    *found_msg_ref = NULL;

    for(item  = opal_list_get_first(&ompi_crcp_bkmrk_pml_peer_refs);
        item != opal_list_get_end(&ompi_crcp_bkmrk_pml_peer_refs);
        item  = opal_list_get_next(item) ) {
        cur_peer_ref = (ompi_crcp_bkmrk_pml_peer_ref_t*)item;

        /*
         * If we ware not MPI_ANY_SOURCE, then extract the process name from the
         * communicator, and search only the peer that matches.
         */
        if( MPI_ANY_SOURCE != peer && peer >= 0) {
            /* Check to see if peer could possibly be in this communicator */
            if( comm->c_local_group->grp_proc_count <= peer ) {
                continue;
            }

            if( OPAL_EQUAL != orte_util_compare_name_fields(ORTE_NS_CMP_ALL,
                                                            &(cur_peer_ref->proc_name),
                                                            &(comm->c_local_group->grp_proc_pointers[peer]->proc_name)) ) {
                continue;
            }
        }

        drain_message_find(&(cur_peer_ref->drained_list),
                           count, tag, peer,
                           comm->c_contextid, ddt_size,
                           found_msg_ref,
                           content_ref);
        if( NULL != *found_msg_ref) {
            if( NULL != peer_ref ) {
                *peer_ref = cur_peer_ref;
            }
            return OMPI_SUCCESS;
        }
    }

    return OMPI_SUCCESS;
}

If peer is MPI_ANY_SOURCE, then drain_message_find_any will fetch a message from any source.
Otherwise, it will look only for messages from the desired peer.

ompi_crcp_bkmrk_pml_peer_refs is populated in ompi/mca/crcp/bkmrk/crcp_bkmrk_pml.c (some lines omitted):

ompi/mca/crcp/bkmrk/crcp_bkmrk_pml.c

/*
 * Copyright (c) 2004-2008 The Trustees of Indiana University.
 *                         All rights reserved.
 * $COPYRIGHT$
 * 
 * Additional copyrights may follow
 * 
 * $HEADER$
 */

/**************** Processes *****************/
ompi_crcp_base_pml_state_t* ompi_crcp_bkmrk_pml_add_procs(
                                   struct ompi_proc_t **procs,
                                   size_t nprocs,
                                   ompi_crcp_base_pml_state_t* pml_state )
{
    int ret;
    ompi_crcp_bkmrk_pml_peer_ref_t *new_peer_ref;
    size_t i;

    if( OMPI_CRCP_PML_PRE != pml_state->state ){
        goto DONE;
    }

    OPAL_OUTPUT_VERBOSE((30, mca_crcp_bkmrk_component.super.output_handle,
                        "crcp:bkmrk: pml_add_procs()"));

    /*
     * Save pointers to the wrapped PML
     */
    wrapped_pml_component = pml_state->wrapped_pml_component;
    wrapped_pml_module    = pml_state->wrapped_pml_module;

    /*
     * Create a peer_ref for each peer added
     */
    for( i = 0; i < nprocs; ++i) {
        HOKE_PEER_REF_ALLOC(new_peer_ref, ret);

        new_peer_ref->proc_name.jobid  = procs[i]->proc_name.jobid;
        new_peer_ref->proc_name.vpid   = procs[i]->proc_name.vpid;

        opal_list_append(&ompi_crcp_bkmrk_pml_peer_refs, &(new_peer_ref->super));
    }

 DONE:
    pml_state->error_code = OMPI_SUCCESS;
    return pml_state;
}

Presumably, nprocs is the number of MPI ranks in MPI_COMM_WORLD because ompi_crcp_bkmrk_pml_add_procs has
no argument struct ompi_communicator_t* comm.

Let's say that the peer is 255 and that the number of MPI ranks in MPI_COMM_WORLD is 256. In this case,
drain_message_find_any will need to loop over all the peers before calling drain_message_find for the peer 255 !

That is slow but it works, right ?

With MPI_ANY_SOURCE, I suspect message starvation can occur because there is no round-robin for the starting point of the loop -- the code always starts with the first peer.

Accordingly, if all peers send messages at the same rate, peers with lower ranks will see their messages received first.

This leads to starvation of peers with higher ranks.

The VirtualProcessor technology in Ray

2011-08-25T15:09:00.000-07:00

The VirtualProcessor I developed enables any MPI rank to have thousands of workers working
on different tasks. In reality, only one worker can work at any given moment, but
the VirtualProcessor schedules fairly the workers on the only instruction pipeline
available so that is 1d not a problem at all.

The VirtualProcessor is a technology that make thousands of worker compute tasks
in parallel on a single MPI rank. Obviously, only one such worker is active at
any point, but they all get to work.

The idea is that when a worker pushes a message on the VirtualCommunicator, it has
to wait for a reply. And this reply may arrive later. The idea of the
VirtualProcessor is to easily submit communication-intensive tasks.

Basically, the VirtualCommunicator groups smaller messages into larger messages
to send fewer messages on the physical network.

But to achieve that, an easy way of generating a lot of small messages is
needed. This is the use of the VirtualProcessor.

== Implementation ==

Author: Sébastien Boisvert
License: GPL
code/scheduling/VirtualProcessor.h
code/scheduling/VirtualProcessor.cpp

http://github.com/sebhtml/ray

== Workers ==

In Ray, each MPI rank actually has thousands of workers within it. All these
workers are scheduled, one after the other, inside the same process. No thread
are utilised as these would trash the processor cache.

At any point in time, each of these worker is in one of these states:

- active;
- sleeping;
- completed.

active

    The worker is not waiting for a message reply.

sleeping

    The worker is waiting for a message reply.

completed

    The worker has completed its task.

Understand the main loop of message-passing-interface software

2011-07-21T21:55:00.000-07:00

In video games, the main loop usually looks like this:

while(running){
    processInput();
    updateGameState();
    drawScreen();
}

(from higherorderfun.com)

Message-passing-interface (MPI) software can be designed in a similar fashion.

Each message-passing-interface rank of an MPI software has its own message inbox and its own message outbox.

Like for emails, received messages go in the inbox and sent messages go in the outbox.

The main loop of an MPI rank usually looks like this:

while(running){
    receiveMessages();
    processMessages();
    processData();
    sendMessages();
}

This general architecture is utilised in the Ray de novo genome assembler.

Read the code: github.com/sebhtml/ray

I don't understand why Ray was not included in the paper Bioinformatics 27, 2031–2037

2011-07-19T09:30:00.000-07:00

In the paper

Lin, Y. et al. Comparative studies of de novo assembly tools for next-generation sequencing technologies. Bioinformatics 27, 2031–2037 (2011).

Ray is not mentioned.

Why is that so ?
We have been working on Ray for quite a while with an early prototype of the assembly engine called OpenAssembler (started in 2009-01-21).

We published Ray in 2010 in Journal of Computational Biology.

We have also presented Ray at a few places.

Cool facts about Ray:

- We are participating to Assemblathon 2.
- We assembled a genome on 512 compute cores in 18 hours, there were > 3 000 000 000 Illumina TruSeq paired reads (that is a lot of reads !)
- Ray is an high-performance peer-to-peer assembler
- Can assemble mixtures of technologies
- Is open source, licensed with the GNU GPL
- Ray is free.
- Works on POSIX systems (Linux, Mac, and others) and on Microsoft Windows
- Compiles cleanly with gcc and with Microsoft Visual Studio

- Ray utilises the message-passing interface

- Works well with Open-MPI or MPICH2 -- the 2 main open source implementations of the MPI standard.

- Ray does very few assembly errors.

- Ray is a single executable called Ray
- Implemented in C++
- Ray is object-oriented
- Ray is modular
- Ray is scalable

- Ray is easy to use.

- All the code is on github

I think the paper should have compared Ray with the other assemblers...

Sébastien

More on virtual communication with the message-passing interface.

2011-06-13T14:36:00.000-07:00

The message-passing interface (MPI) is a standard that allows numerous computers to communicate in order to achieve a large-scale peer-to-peer communication in a high-performance matter.

OK, let's face it, writing computer programs with MPI can be hard and tedious. However, one can devise a set of techniques that collectively enhance his/her software craftmanship.

I write C++ (1998) using MPI -- you already know that if you read my blog. In ray, I implemented a few tricks to make the message-passing thing a lot easier.

Slave modes and master modes

First, each MPI rank (a rank is usually mapped to a process, if that matters) has a slave mode and a master mode. A slave mode can be called SLAVE_MODE_COUNT_ENTRIES, which would obviously count entries. Master modes follow the same principle. If an MPI rank is in a state such that it must wait for others, than there is this very special slave mode called SLAVE_MODE_DO_NOTHING.

Each of these modes has an associated callback method. SLAVE_MODE_DO_NOTHING's callback method is called

void Machine::call_SLAVE_MODE_DO_NOTHING();

Its implementation is quite simple, actually.

void Machine::call_SLAVE_MODE_DO_NOTHING(){

/* do nothing */

}
OK, that was funny.

Given a slave mode, I want to call its associated callback method. To do so, I use an array of method pointers. Simple enough.

Keep it simple, stupid

But wait, for each slave mode (the same is true for master modes), I have to

1. Add it in a enumeration so it can be used later;
2. Define its string (read char*) representation for debugging purposes.;
3. Define its callback method prototype (in .h);
4. Define its callback method definition (in .cpp);
5. Add its callback method pointer in the array of method pointers.

Accordingly, to add a new slave mode, I would need to add things at at least 5 places. To avoid that, I use macros.

An example follows.

core/slave_mode_macros.h

MACRO_LIST_ITEM( RAY_SLAVE_MODE_LOAD_SEQUENCES )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_START_SEEDING )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_DO_NOTHING )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_SEND_EXTENSION_DATA )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_ASSEMBLE_WAVES )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_FUSION )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_FINISH_FUSIONS )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_DISTRIBUTE_FUSIONS )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_AMOS )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_AUTOMATIC_DISTANCE_DETECTION )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_SEND_LIBRARY_DISTANCES )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_EXTRACT_VERTICES )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_SEND_DISTRIBUTION )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_EXTENSION )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_INDEX_SEQUENCES )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_REDUCE_MEMORY_CONSUMPTION )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_DELETE_VERTICES )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_SEND_SEED_LENGTHS )
MACRO_LIST_ITEM( RAY_SLAVE_MODE_SCAFFOLDER )

1. Add it in a enumeration so it can be used later

core/slave_modes.h

#define MACRO_LIST_ITEM(element) element,

enum{
#include "core/slave_mode_macros.h"
SLAVE_MODES_H_DUMMY
};

#undef MACRO_LIST_ITEM

2. Define its string (read char*) representation for debugging purposes.

core/slave_modes.cpp

#define MACRO_LIST_ITEM(x) #x,

const char* SLAVE_MODES[]={
#include "core/slave_mode_macros.h"
};

#undef MACRO_LIST_ITEM

3. Define its callback method prototype (in .h)

core/Machine.h

#define MACRO_LIST_ITEM(x) void call_ ## x();
#include "core/slave_mode_macros.h"
#undef MACRO_LIST_ITEM

5. Add its callback method pointer in the array of method pointers

core/Machine.cpp

#define MACRO_LIST_ITEM(x) m_slave_methods[x]=&Machine::call_ ## x ;
#include "core/slave_mode_macros.h"
#undef MACRO_LIST_ITEM

Note that the definition of a callback is not done with these macros because it can not be automated.

The same principle is applied to master modes and to MPI message types. A callback method for an MPI message type receives a message and processes it.

Wary workers

Let us say that I want to compute the following function.

int sum(){

int sum=0;

int i=1000;

while(i--){

sum+=i;

}

return sum;

}

Now, let's do it the worker's way.

The methods of a worker usually are:

void Worker::work();
bool Worker::isDone();
Result Worker::getWorkResult();

In my case:

class Worker{

int m_i;

int m_sum;

bool m_done;

public:

void constructor(){

m_i=10000;

m_sum=0;

m_done=false;

}

void work{

if(m_done){

return;

}

m_sum+=m_i;

m_i--;

if(m_i==0){

m_done=true;

}

bool isDone(){

return m_done;

}

int getWorkResult(){

return m_sum;

}

};

I can now use this worker to compute function sum:

int sumWithWorker(){

Worker worker;

worker.constructor();

while(!worker.isDone()){

worker.work();

}

return worker.getWorkResult();

}

This allows the computation to be sliced in parts. This is handy for programming an MPI application because it produces naturally fine granularity. By the way, fine granularity is necessary to obtain low-latency message passing. Otherwise, an MPI rank may have to wait several minutes before receiving its response -- which is perhaps not very acceptable.

Proper message passing and virtual communication

Now, the most time-consuming part of programming with MPI: message transiting -- from one end, we send a message to a destination rank. On the other end, we receive the message. It can be tedious to do this message-passing recipe and obviously you don't want to use a spinlock while waiting for a message's response. Instead, you want to be able to receive messages from other MPI ranks.

In a previous post, I described the virtual communicator. Here, I use the concept again.

Given a piece of MPI code, the 3 basic necessary operations from a worker's point of view are

- void VirtualCommunicator::pushMessage(uint64_t workerId,Message*message);
- bool VirtualCommunicator::isMessageProcessed(uint64_t workerId);
- vector VirtualCommunicator::getMessageResponseElements(uint64_t workerId);

Typical workflow:

The worker identified by workerId pushes a message to the virtual communicator.

The worker identified by workerId asks the virtual communicator if its message has been processed.

If so, the worker identifier by workerId asks its message response to the virtual communicator.

Now, we don't want to send numerous skinny messages (one for each call to pushMessage). Instead, automatic message aggregation is highly desired.
In ray, the VirtualCommunicator does just that right away, for you.

To achieve automatic message aggregation, a pool of workers is needed. Each of these workers are independent, but their messages will be grouped without them knowing it.

This method alone yields a significant performance gain.

What follows is an gain example.

Rank 14 VirtualCommunicator Statistics

Rank 14: 60466 virtual messages for 20854019 pushed messages

Rank 14: percentage= 0.289949%

Out of the 20854019 pushed messages, only 60466 virtual messages were sent.

Berserk producers overflow ring buffers

Note that a worker never goes berserk -- it never sends numerous messages without first getting a response for its first pushed message. Thus, no problem of consumer and producer can occur.

gg

sebhtml

de novo assembly of Illumina CEO genome in 11.5 h

2011-03-21T09:20:00.000-07:00

THE initial aims of our group regarding de novo assembly of genomes were:

1. To assemble genomes using mixes of sequencing technologies simultaneously.
2. To assemble large repeat-rich genomes.
3. To devise novel approaches to deal with repeats.

1. To assemble genomes using mixes of sequencing technologies simultaneously.

We showed the feasibility of using mixes of sequencing technologies simultaneously using Ray -- a de novo genome assembler -- see Journal of Computational Biology 17(11): 1519-1533. Ray follows the Single Program, Multiple Data approach. It is implemented using the Message-Passing Interface.

With this method, a computation is separated into data-dependent parts. Each part is given to a processor and any processor communicates with others to access remote information.

2. To assemble large repeat-rich genomes.

It was thought that message transit in Ray would interfere strongly with the feasibility of genome assembly of large repeat-rich genomes.

On 12th of December 2010, I had written an entry on resource consumption of Ray. In the same writings, only the sheer memory usage was reported -- no computation time was available as it was still unclear it would be feasible to use Message-Passing Interface end-to-end. The k-mer length used was 19. This low k-mer value did not allow Ray to produce significant assemblies.

Since then, I have worked on the in-memory data storage systems, refined manual message aggregation systems, and further investigated ways of automatically grouping communications. One approach that I developed that effectively accelerates the computation was the design, implementation and integration of a virtual communicator on the top of MPI's default communicator -- MPI_COMM_WORLD. For an overview of the Virual Communicator, read Arnie's story.

On the 1st of January 2011, our group was awarded a special allocation resource project from Compute Canada. However, we gained access to this resource only one month later: on the 4th of February 2011.

I wandered on the Internet and fetched the genome data of the CEO of Illumina (SRA010766). These are large numbers: 6 372 129 288 reads, each in pair and having 75 nucleotides. The bits sum to 477 909 696 600 nucleotides.

Here, I report a de novo assembly of Illumina CEO genome in 11.5 hours using a supercomputer (on Compute Canada's colosse; Job identifier: 2814556; code name: Nitro). The computation was done with Ray 1.3.0 and the k-mer length was 21 (default value in Ray).

Ray detected pairs with an average outer distance equal to 190 and a standard deviation of 30. The peak coverage was 22 and the minimum coverage was 6. There were 1 803 534 contiguous sequences with at least 100 nucleotides, with a total of 1 772 120 417 nucleotides. The N50 was 1341 and the average length was 982. Finally, the length of the longest contiguous sequence was 14584.

Table 1: Running time of Ray on genome data of the CEO of Illumina.

Algorithm step	Elapsed time
Distribution of sequence reads	42 minutes, 1 seconds
Distribution of vertices & edges	18 minutes, 36 seconds
Calculation of coverage distribution	19 seconds
Indexing of sequence reads	24 minutes, 42 seconds
Computation of seeds	16 minutes, 14 seconds
Computation of library sizes	2 minutes, 58 seconds
Extension of seeds	7 hours, 10 minutes, 38 seconds
Computation of fusions	2 hours, 26 minutes, 26 seconds
Collection of fusions	1 minutes, 46 seconds
Total	11 hours, 23 minutes, 40 seconds

64 computers were utilised. Each had 24 GiB of physical memory and 2 Intel Nehalem-EP processors. Each processor had 4 compute cores. There were a total of 512 compute cores and 1536 GiB of distributed physical memory. Computer were interconnected with InfiniBand QDR (40 Gigabits per second).

For people that don't have access to a supercomputer nearby, compute time can be rented from compute cloud providers. An example is Amazon Elastic Cloud Compute (EC2). Ray is cloud-ready, so to say. The Amazon EC2 Cluster Compute provides low latency, full bisection 10 Gigabits per second bandwidth.

3. To devise novel approaches to deal with repeats.

In addition, I developped novel ways of enabling repeat traversal with paired sequences. I will present this work during the First Annual RECOMB Satellite Workshop on Massively Parallel Sequencing. The 15-minute talk's title is Constrained traversal of repeats with paired sequences and will take place on 27 March 2011 from 12:10 to 12:30.

Perspective

* To try a larger k-mer length.
* To assemble the Yoruban African Genome (SRA000271).

Ray is freely (librement) available on Sourceforge.

The Ray paper is freely accessible:

Ray: simultaneous assembly of reads from a mix of high-throughput sequencing technologies.
Sébastien Boisvert, François Laviolette, and Jacques Corbeil.
Journal of Computational Biology (Mary Ann Liebert, Inc. publishers).
November 2010, 17(11): 1519-1533.
doi:10.1089/cmb.2009.0238

Edited on March 21st, 2011 to correct a typographical error.

The Virtual Communicator

2011-01-23T17:21:00.001-08:00

IT WAS a wintry day of January, in a coldly-tempered land. On this island lived peculiar citizens whose main everyday whereabouts involved producing goods and getting back some credits for the hard work. Arnie was one of them. He was cultivating vegetables, in a greenhouse in the cold season, obviously. He would prepare and send a shipment whenever his plums were ready. But most often his daily production, which departed his ranch daily, did not fill all of the available space provided by the Raw Communicator. The Raw Communicator is the entity whose personhood solely involves delivering the production of a worker's goods to the markets, and bringing back the gained assets to the farmer. And because he was so raw, the Raw Communicator would not even care of visiting other folks after visiting Arnie. He would directly transit to the markets. Afterwards, he would visit Arnie's first neighbour, return to the market, and so on.

Arnie, although being just a potent grower of greens, felt that the Raw Communicator should be less raw and more genuinely efficient. Arnie had brought up the matter to the Raw Communicator, but to no avail. The latter had stated that his raw nature prohibited him from doing anything else from what he was here to do: delivering goods to the markets and bringing back good fortune to the goods's erstwhile owner.

Just like he felt, there was many of his kind who wished communications were faster -- they were insulted by the delays brought by the way of message transportation. And they would not secure a way of grouping their parcels -- before sending them -- by speaking to each other; that would be utterly called communism in this sacred freedom-driven world. Although they knew that the transportation mechanism could hold a particular amount of goods -- an amount being upper-bounded by some arcane physic laws – they held onto their traditions of not gathering their deeds together and instead continued to ship them individually, one by one in these harsh wintering times. What was going on, in fact, was simply an unwillingness to modify their habits. What they needed was a novel mechanism under which they could continue their whereabouts without changing their customs. Should this almighty desired interface be available, the earthlings would only want to send their package to a destination, and to receive a timely response from that very said destination. Less is more. No more, no less.

Then arrived an opportunistic celebrity. His name was Virtual Communicator. And he said on his arrival:

“Virtual Communicator, that is my name. Oh ! Honourable citizens of this spectacular communicative gateway, not only could your population save energy by grouping your merchandise, but your collective wisdom would also save precious time should you select me as your communicator. Yes ! I say drop your vanilla communicator, the Raw Communicator is outdated, rather slow and quite expensive !”

This newly-arrived character then went on:

“My ways are the same for you, but I can group bits and packets to make you save some gas and minerals. All I ask is a mere percentage of your planned financial gains as predictable by scheduled grouping of your workers's production.”

Arnie and his fellow neighbours listened carefully -- they truly thought that the Virtual Communicator sounded like an huge improvement from the Raw Communicator. Being a spectator of these reactions, the Virtual Communicator resumed talking:

“In the end, you can send and receive letters faster, the process saves your energy and your time. You will also gain considerable financial advantages. Should you assign me to this gracious task, I would also be granted an occupation with associated remuneration.”

But the folks from the cold city wanted to know the motivation of this recently introduced entity. The Virtual Communicator then explained his motives.

“Yes, I the great Virtual Communicator am a hub -- a gateway. I make profits from work others perform. These workers -- who produce the goods and needs -- are however wordless when it comes to leveraging efficiency and distribution. Needless to say, I the great Virtual Communicator can't do work like worker Arnie does. And neither Arnie can do my job. Pals, it is clearly a win-win situation here.”

“What about the Raw Communicator ?” asked Arnie. “He too deserve occupational remuneration” he added. The Virtual Communicator then explained that he was not able to actually send and receive merchandises, that all he could do was grouping and ungrouping things. The rest was part of the personhood of the Raw Communicator. To put it simply, he explained that the Raw Communicator actually receives credit on a per-message basis. Although he would notice a reduction of messaging throughput, the state-sponsored part of his salary would remain a frozen asset for him nonetheless.

“What do you say ?” asked finally the Virtual Communicator.

I wrote this story while working on virtual (read abstract) message-grouping in a message-passing-interface environment.

Ray and large datasets

2010-12-12T09:28:00.000-08:00

Ray is a de novo de Bruijn genome assembler that uses message passing interface. Sébastien Boisvert reports the memory utilization for Illumina CEO Jay T. Flatley (SRA010766) with k=19.

The short read archive hosts several large sequencing datasets. Among these, there are SRA000271 and SRA010766 -- the two are reads from human genomes: the first is an African male individual (HapMap: NA18507) and the other is Illumina CEO Jay T. Flatley.

Variations in the genome of these individuals can be harnessed by mapping the reads onto a sequence reference -- a set of files containing the genetic blueprint as determined by the investigators of the human genome project.

But when a reference is not available, de novo assembly of the sequencing reads into contiguous sequences is one of the first step toward acquiring an high-quality reference sequence.

Unparalleled parallelism

In parallel computation, powerful processors can be combined to form an alliance stronger and faster at analyzing problems. One way to construct such computation facilities is to interconnect many computers, and use a paradigm known as message passing interface wherein messages are sent between compute nodes to interleave computation and communication.

To the best of my knowledge, de novo assemblers that use the message passing interface are few: ABySS and Ray. ABySS is a set of programs glued together with make files whereas everything is embedded in one program in Ray.

This makes Ray straightforward to launch on super computers or in the cloud -- using schedulers such as Sun Grid Engine. Ray is compliant with the standard MPI 2.2 and is free/open source software.

SRA010766: Illumina CEO data

There are 164 sequence files in this dataset. Paired reads (note the suffixes _1 and _2) in this dataset have a length of 75.

Number of sequences in the dataset SRA010766

File number	File name	Number of sequences	Sequence length	Number of nucleotides
1	SRR033552_1.fastq	40966220	75	3072466500
2	SRR033552_2.fastq	40966220	75	3072466500
3	SRR033553_1.fastq	43599280	75	3269946000
4	SRR033553_2.fastq	43599280	75	3269946000
5	SRR033554_1.fastq	41531088	75	3114831600
6	SRR033554_2.fastq	41531088	75	3114831600
7	SRR033555_1.fastq	42828160	75	3212112000
8	SRR033555_2.fastq	42828160	75	3212112000
9	SRR033556_1.fastq	46543048	75	3490728600
10	SRR033556_2.fastq	46543048	75	3490728600
11	SRR033557_1.fastq	46074808	75	3455610600
12	SRR033557_2.fastq	46074808	75	3455610600
13	SRR033558_1.fastq	42524412	75	3189330900
14	SRR033558_2.fastq	42524412	75	3189330900
15	SRR033559_1.fastq	43244880	75	3243366000
16	SRR033559_2.fastq	43244880	75	3243366000
17	SRR033560_1.fastq	44682732	75	3351204900
18	SRR033560_2.fastq	44682732	75	3351204900
19	SRR033561_1.fastq	35273964	75	2645547300
20	SRR033561_2.fastq	35273964	75	2645547300
21	SRR033562_1.fastq	34751780	75	2606383500
22	SRR033562_2.fastq	34751780	75	2606383500
23	SRR033563_1.fastq	35911452	75	2693358900
24	SRR033563_2.fastq	35911452	75	2693358900
25	SRR033564_1.fastq	34959916	75	2621993700
26	SRR033564_2.fastq	34959916	75	2621993700
27	SRR033565_1.fastq	33651596	75	2523869700
28	SRR033565_2.fastq	33651596	75	2523869700
29	SRR033566_1.fastq	27935216	75	2095141200
30	SRR033566_2.fastq	27935216	75	2095141200
31	SRR033567_1.fastq	23905716	75	1792928700
32	SRR033567_2.fastq	23905716	75	1792928700
33	SRR033568_1.fastq	21055424	75	1579156800
34	SRR033568_2.fastq	21055424	75	1579156800
35	SRR033569_1.fastq	54777660	75	4108324500
36	SRR033569_2.fastq	54777660	75	4108324500
37	SRR033570_1.fastq	53258444	75	3994383300
38	SRR033570_2.fastq	53258444	75	3994383300
39	SRR033571_1.fastq	54960632	75	4122047400
40	SRR033571_2.fastq	54960632	75	4122047400
41	SRR033572_1.fastq	52316296	75	3923722200
42	SRR033572_2.fastq	52316296	75	3923722200
43	SRR033573_1.fastq	39423228	75	2956742100
44	SRR033573_2.fastq	39423228	75	2956742100
45	SRR033574_1.fastq	38145116	75	2860883700
46	SRR033574_2.fastq	38145116	75	2860883700
47	SRR033575_1.fastq	43014116	75	3226058700
48	SRR033575_2.fastq	43014116	75	3226058700
49	SRR033576_1.fastq	43881552	75	3291116400
50	SRR033576_2.fastq	43881552	75	3291116400
51	SRR033577_1.fastq	44275780	75	3320683500
52	SRR033577_2.fastq	44275780	75	3320683500
53	SRR033578_1.fastq	50932256	75	3819919200
54	SRR033578_2.fastq	50932256	75	3819919200
55	SRR033579_1.fastq	50963792	75	3822284400
56	SRR033579_2.fastq	50963792	75	3822284400
57	SRR033580_1.fastq	47329708	75	3549728100
58	SRR033580_2.fastq	47329708	75	3549728100
59	SRR033581_1.fastq	36354620	75	2726596500
60	SRR033581_2.fastq	36354620	75	2726596500
61	SRR033582_1.fastq	42518488	75	3188886600
62	SRR033582_2.fastq	42518488	75	3188886600
63	SRR033583_1.fastq	40551820	75	3041386500
64	SRR033583_2.fastq	40551820	75	3041386500
65	SRR033584_1.fastq	41197920	75	3089844000
66	SRR033584_2.fastq	41197920	75	3089844000
67	SRR033585_1.fastq	38647136	75	2898535200
68	SRR033585_2.fastq	38647136	75	2898535200
69	SRR033586_1.fastq	37941964	75	2845647300
70	SRR033586_2.fastq	37941964	75	2845647300
71	SRR033587_1.fastq	45184196	75	3388814700
72	SRR033587_2.fastq	45184196	75	3388814700
73	SRR033588_1.fastq	49829552	75	3737216400
74	SRR033588_2.fastq	49829552	75	3737216400
75	SRR033589_1.fastq	49263392	75	3694754400
76	SRR033589_2.fastq	49263392	75	3694754400
77	SRR033590_1.fastq	46582328	75	3493674600
78	SRR033590_2.fastq	46582328	75	3493674600
79	SRR033591_1.fastq	49959800	75	3746985000
80	SRR033591_2.fastq	49959800	75	3746985000
81	SRR033592_1.fastq	49806624	75	3735496800
82	SRR033592_2.fastq	49806624	75	3735496800
83	SRR033593_1.fastq	44973264	75	3372994800
84	SRR033593_2.fastq	44973264	75	3372994800
85	SRR033594_1.fastq	35298140	75	2647360500
86	SRR033594_2.fastq	35298140	75	2647360500
87	SRR033595_1.fastq	33396224	75	2504716800
88	SRR033595_2.fastq	33396224	75	2504716800
89	SRR033596_1.fastq	33513072	75	2513480400
90	SRR033596_2.fastq	33513072	75	2513480400
91	SRR033597_1.fastq	33738536	75	2530390200
92	SRR033597_2.fastq	33738536	75	2530390200
93	SRR033598_1.fastq	33698860	75	2527414500
94	SRR033598_2.fastq	33698860	75	2527414500
95	SRR033599_1.fastq	34741596	75	2605619700
96	SRR033599_2.fastq	34741596	75	2605619700
97	SRR033600_1.fastq	35548768	75	2666157600
98	SRR033600_2.fastq	35548768	75	2666157600
99	SRR033601_1.fastq	30653280	75	2298996000
100	SRR033601_2.fastq	30653280	75	2298996000
101	SRR033602_1.fastq	31295988	75	2347199100
102	SRR033602_2.fastq	31295988	75	2347199100
103	SRR033603_1.fastq	22230280	75	1667271000
104	SRR033603_2.fastq	22230280	75	1667271000
105	SRR033604_1.fastq	22552392	75	1691429400
106	SRR033604_2.fastq	22552392	75	1691429400
107	SRR033605_1.fastq	22204860	75	1665364500
108	SRR033605_2.fastq	22204860	75	1665364500
109	SRR033606_1.fastq	24272588	75	1820444100
110	SRR033606_2.fastq	24272588	75	1820444100
111	SRR033607_1.fastq	25241752	75	1893131400
112	SRR033607_2.fastq	25241752	75	1893131400
113	SRR033608_1.fastq	25934688	75	1945101600
114	SRR033608_2.fastq	25934688	75	1945101600
115	SRR033609_1.fastq	29209040	75	2190678000
116	SRR033609_2.fastq	29209040	75	2190678000
117	SRR033610_1.fastq	30452744	75	2283955800
118	SRR033610_2.fastq	30452744	75	2283955800
119	SRR033611_1.fastq	33816544	75	2536240800
120	SRR033611_2.fastq	33816544	75	2536240800
121	SRR033612_1.fastq	34501164	75	2587587300
122	SRR033612_2.fastq	34501164	75	2587587300
123	SRR033613_1.fastq	29660620	75	2224546500
124	SRR033613_2.fastq	29660620	75	2224546500
125	SRR033614_1.fastq	37914656	75	2843599200
126	SRR033614_2.fastq	37914656	75	2843599200
127	SRR033615_1.fastq	38005976	75	2850448200
128	SRR033615_2.fastq	38005976	75	2850448200
129	SRR033616_1.fastq	43965744	75	3297430800
130	SRR033616_2.fastq	43965744	75	3297430800
131	SRR033617_1.fastq	46335384	75	3475153800
132	SRR033617_2.fastq	46335384	75	3475153800
133	SRR033618_1.fastq	46948232	75	3521117400
134	SRR033618_2.fastq	46948232	75	3521117400
135	SRR033619_1.fastq	47209944	75	3540745800
136	SRR033619_2.fastq	47209944	75	3540745800
137	SRR033620_1.fastq	45119616	75	3383971200
138	SRR033620_2.fastq	45119616	75	3383971200
139	SRR033621_1.fastq	26939464	75	2020459800
140	SRR033621_2.fastq	26939464	75	2020459800
141	SRR033622_1.fastq	25720336	75	1929025200
142	SRR033622_2.fastq	25720336	75	1929025200
143	SRR033623_1.fastq	27676576	75	2075743200
144	SRR033623_2.fastq	27676576	75	2075743200
145	SRR033624_1.fastq	31982932	75	2398719900
146	SRR033624_2.fastq	31982932	75	2398719900
147	SRR033625_1.fastq	35308472	75	2648135400
148	SRR033625_2.fastq	35308472	75	2648135400
149	SRR033626_1.fastq	35703328	75	2677749600
150	SRR033626_2.fastq	35703328	75	2677749600
151	SRR033627_1.fastq	33266868	75	2495015100
152	SRR033627_2.fastq	33266868	75	2495015100
153	SRR033628_1.fastq	53547956	75	4016096700
154	SRR033628_2.fastq	53547956	75	4016096700
155	SRR033629_1.fastq	45474980	75	3410623500
156	SRR033629_2.fastq	45474980	75	3410623500
157	SRR033630_1.fastq	40327452	75	3024558900
158	SRR033630_2.fastq	40327452	75	3024558900
159	SRR033631_1.fastq	41158436	75	3086882700
160	SRR033631_2.fastq	41158436	75	3086882700
161	SRR033632_1.fastq	54755312	75	4106648400
162	SRR033632_2.fastq	54755312	75	4106648400
163	SRR033633_1.fastq	41212468	75	3090935100
164	SRR033633_2.fastq	41212468	75	3090935100
Total	-	6372129288	-	477909696600

Assuming the genome size is 3 gigabases, the coverage is 159.

Memory utilization

To run and develop Ray on this large dataset, I utilize colosse: 960 compute nodes each having two quad-core Intel Nehalem-EP -- 8 cores -- and 24 gigabytes of memory. I runned Ray on 64 computers: 512 processors cores and 1536 gigabytes of distributed memory.


Global identifier	Per-host identifier	Host-specific queue-name	Type	Job name	Job identifier	Host name	Average load	Number of processor cores	Host-specific utilized memory	Host-specific available memory
1	1	mpi@r104-n27	MASTER	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
1	1	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
2	2	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
3	3	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
4	4	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
5	5	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
6	6	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
7	7	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
8	8	mpi@r104-n27	SLAVE	Ray.r4140	1184258	r104-n27	8.20	8	20.4G	23.5G
9	1	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
10	2	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
11	3	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
12	4	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
13	5	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
14	6	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
15	7	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
16	8	mpi@r105-n49	SLAVE	Ray.r4140	1184258	r105-n49	8.10	8	20.2G	23.5G
17	1	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
18	2	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
19	3	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
20	4	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
21	5	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
22	6	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
23	7	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
24	8	mpi@r105-n51	SLAVE	Ray.r4140	1184258	r105-n51	8.14	8	20.1G	23.5G
25	1	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
26	2	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
27	3	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
28	4	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
29	5	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
30	6	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
31	7	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
32	8	mpi@r106-n27	SLAVE	Ray.r4140	1184258	r106-n27	8.04	8	20.6G	23.5G
33	1	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
34	2	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
35	3	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
36	4	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
37	5	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
38	6	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
39	7	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
40	8	mpi@r106-n29	SLAVE	Ray.r4140	1184258	r106-n29	8.13	8	20.1G	23.5G
41	1	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
42	2	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
43	3	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
44	4	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
45	5	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
46	6	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
47	7	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
48	8	mpi@r106-n35	SLAVE	Ray.r4140	1184258	r106-n35	8.00	8	20.6G	23.5G
49	1	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
50	2	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
51	3	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
52	4	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
53	5	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
54	6	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
55	7	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
56	8	mpi@r106-n36	SLAVE	Ray.r4140	1184258	r106-n36	8.00	8	20.8G	23.5G
57	1	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
58	2	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
59	3	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
60	4	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
61	5	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
62	6	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
63	7	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
64	8	mpi@r106-n41	SLAVE	Ray.r4140	1184258	r106-n41	8.00	8	20.2G	23.5G
65	1	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
66	2	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
67	3	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
68	4	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
69	5	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
70	6	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
71	7	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
72	8	mpi@r106-n50	SLAVE	Ray.r4140	1184258	r106-n50	8.20	8	20.0G	23.5G
73	1	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
74	2	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
75	3	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
76	4	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
77	5	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
78	6	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
79	7	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
80	8	mpi@r106-n60	SLAVE	Ray.r4140	1184258	r106-n60	8.00	8	20.4G	23.5G
81	1	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
82	2	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
83	3	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
84	4	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
85	5	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
86	6	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
87	7	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
88	8	mpi@r106-n61	SLAVE	Ray.r4140	1184258	r106-n61	10.00	8	21.3G	23.5G
89	1	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
90	2	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
91	3	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
92	4	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
93	5	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
94	6	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
95	7	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
96	8	mpi@r106-n69	SLAVE	Ray.r4140	1184258	r106-n69	8.00	8	20.0G	23.5G
97	1	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
98	2	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
99	3	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
100	4	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
101	5	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
102	6	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
103	7	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
104	8	mpi@r107-n52	SLAVE	Ray.r4140	1184258	r107-n52	8.03	8	20.1G	23.5G
105	1	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
106	2	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
107	3	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
108	4	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
109	5	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
110	6	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
111	7	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
112	8	mpi@r107-n73	SLAVE	Ray.r4140	1184258	r107-n73	8.12	8	20.6G	23.5G
113	1	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
114	2	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
115	3	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
116	4	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
117	5	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
118	6	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
119	7	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
120	8	mpi@r107-n74	SLAVE	Ray.r4140	1184258	r107-n74	8.11	8	20.4G	23.5G
121	1	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
122	2	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
123	3	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
124	4	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
125	5	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
126	6	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
127	7	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
128	8	mpi@r107-n77	SLAVE	Ray.r4140	1184258	r107-n77	8.12	8	20.6G	23.5G
129	1	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
130	2	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
131	3	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
132	4	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
133	5	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
134	6	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
135	7	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
136	8	mpi@r107-n78	SLAVE	Ray.r4140	1184258	r107-n78	8.13	8	20.6G	23.5G
137	1	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
138	2	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
139	3	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
140	4	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
141	5	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
142	6	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
143	7	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
144	8	mpi@r107-n88	SLAVE	Ray.r4140	1184258	r107-n88	8.02	8	20.1G	23.5G
145	1	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
146	2	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
147	3	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
148	4	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
149	5	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
150	6	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
151	7	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
152	8	mpi@r107-n92	SLAVE	Ray.r4140	1184258	r107-n92	8.15	8	20.4G	23.5G
153	1	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
154	2	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
155	3	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
156	4	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
157	5	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
158	6	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
159	7	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
160	8	mpi@r108-n35	SLAVE	Ray.r4140	1184258	r108-n35	8.00	8	20.5G	23.5G
161	1	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
162	2	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
163	3	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
164	4	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
165	5	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
166	6	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
167	7	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
168	8	mpi@r108-n40	SLAVE	Ray.r4140	1184258	r108-n40	8.13	8	20.0G	23.5G
169	1	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
170	2	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
171	3	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
172	4	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
173	5	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
174	6	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
175	7	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
176	8	mpi@r108-n41	SLAVE	Ray.r4140	1184258	r108-n41	8.13	8	20.1G	23.5G
177	1	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
178	2	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
179	3	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
180	4	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
181	5	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
182	6	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
183	7	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
184	8	mpi@r108-n47	SLAVE	Ray.r4140	1184258	r108-n47	8.00	8	20.5G	23.5G
185	1	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
186	2	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
187	3	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
188	4	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
189	5	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
190	6	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
191	7	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
192	8	mpi@r108-n48	SLAVE	Ray.r4140	1184258	r108-n48	8.00	8	20.4G	23.5G
193	1	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
194	2	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
195	3	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
196	4	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
197	5	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
198	6	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
199	7	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
200	8	mpi@r108-n59	SLAVE	Ray.r4140	1184258	r108-n59	8.15	8	20.8G	23.5G
201	1	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
202	2	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
203	3	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
204	4	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
205	5	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
206	6	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
207	7	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
208	8	mpi@r108-n64	SLAVE	Ray.r4140	1184258	r108-n64	8.00	8	20.5G	23.5G
209	1	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
210	2	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
211	3	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
212	4	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
213	5	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
214	6	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
215	7	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
216	8	mpi@r108-n77	SLAVE	Ray.r4140	1184258	r108-n77	8.00	8	21.0G	23.5G
217	1	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
218	2	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
219	3	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
220	4	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
221	5	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
222	6	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
223	7	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
224	8	mpi@r108-n84	SLAVE	Ray.r4140	1184258	r108-n84	8.02	8	20.6G	23.5G
225	1	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
226	2	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
227	3	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
228	4	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
229	5	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
230	6	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
231	7	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
232	8	mpi@r108-n91	SLAVE	Ray.r4140	1184258	r108-n91	8.06	8	20.7G	23.5G
233	1	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
234	2	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
235	3	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
236	4	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
237	5	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
238	6	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
239	7	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
240	8	mpi@r109-n1	SLAVE	Ray.r4140	1184258	r109-n1	8.14	8	20.4G	23.5G
241	1	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
242	2	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
243	3	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
244	4	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
245	5	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
246	6	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
247	7	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
248	8	mpi@r109-n10	SLAVE	Ray.r4140	1184258	r109-n10	8.11	8	20.5G	23.5G
249	1	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
250	2	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
251	3	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
252	4	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
253	5	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
254	6	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
255	7	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
256	8	mpi@r109-n12	SLAVE	Ray.r4140	1184258	r109-n12	8.01	8	20.2G	23.5G
257	1	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
258	2	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
259	3	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
260	4	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
261	5	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
262	6	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
263	7	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
264	8	mpi@r109-n13	SLAVE	Ray.r4140	1184258	r109-n13	8.13	8	20.6G	23.5G
265	1	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
266	2	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
267	3	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
268	4	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
269	5	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
270	6	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
271	7	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
272	8	mpi@r109-n14	SLAVE	Ray.r4140	1184258	r109-n14	8.00	8	20.5G	23.5G
273	1	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
274	2	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
275	3	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
276	4	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
277	5	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
278	6	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
279	7	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
280	8	mpi@r109-n15	SLAVE	Ray.r4140	1184258	r109-n15	8.17	8	20.1G	23.5G
281	1	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
282	2	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
283	3	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
284	4	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
285	5	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
286	6	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
287	7	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
288	8	mpi@r109-n16	SLAVE	Ray.r4140	1184258	r109-n16	8.01	8	20.5G	23.5G
289	1	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
290	2	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
291	3	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
292	4	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
293	5	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
294	6	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
295	7	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
296	8	mpi@r109-n18	SLAVE	Ray.r4140	1184258	r109-n18	8.00	8	20.5G	23.5G
297	1	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
298	2	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
299	3	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
300	4	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
301	5	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
302	6	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
303	7	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
304	8	mpi@r109-n19	SLAVE	Ray.r4140	1184258	r109-n19	8.14	8	20.6G	23.5G
305	1	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
306	2	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
307	3	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
308	4	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
309	5	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
310	6	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
311	7	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
312	8	mpi@r109-n21	SLAVE	Ray.r4140	1184258	r109-n21	8.16	8	20.3G	23.5G
313	1	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
314	2	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
315	3	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
316	4	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
317	5	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
318	6	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
319	7	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
320	8	mpi@r109-n22	SLAVE	Ray.r4140	1184258	r109-n22	8.00	8	20.6G	23.5G
321	1	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
322	2	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
323	3	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
324	4	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
325	5	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
326	6	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
327	7	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
328	8	mpi@r109-n23	SLAVE	Ray.r4140	1184258	r109-n23	8.13	8	20.5G	23.5G
329	1	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
330	2	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
331	3	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
332	4	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
333	5	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
334	6	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
335	7	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
336	8	mpi@r109-n3	SLAVE	Ray.r4140	1184258	r109-n3	8.00	8	20.5G	23.5G
337	1	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
338	2	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
339	3	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
340	4	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
341	5	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
342	6	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
343	7	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
344	8	mpi@r109-n4	SLAVE	Ray.r4140	1184258	r109-n4	8.05	8	20.6G	23.5G
345	1	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
346	2	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
347	3	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
348	4	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
349	5	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
350	6	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
351	7	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
352	8	mpi@r109-n40	SLAVE	Ray.r4140	1184258	r109-n40	8.06	8	20.4G	23.5G
353	1	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
354	2	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
355	3	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
356	4	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
357	5	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
358	6	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
359	7	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
360	8	mpi@r109-n43	SLAVE	Ray.r4140	1184258	r109-n43	8.00	8	21.0G	23.5G
361	1	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
362	2	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
363	3	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
364	4	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
365	5	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
366	6	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
367	7	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
368	8	mpi@r109-n5	SLAVE	Ray.r4140	1184258	r109-n5	8.00	8	20.5G	23.5G
369	1	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
370	2	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
371	3	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
372	4	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
373	5	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
374	6	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
375	7	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
376	8	mpi@r109-n50	SLAVE	Ray.r4140	1184258	r109-n50	8.14	8	20.2G	23.5G
377	1	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
378	2	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
379	3	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
380	4	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
381	5	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
382	6	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
383	7	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
384	8	mpi@r109-n52	SLAVE	Ray.r4140	1184258	r109-n52	8.00	8	20.3G	23.5G
385	1	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
386	2	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
387	3	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
388	4	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
389	5	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
390	6	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
391	7	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
392	8	mpi@r109-n58	SLAVE	Ray.r4140	1184258	r109-n58	8.02	8	20.6G	23.5G
393	1	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
394	2	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
395	3	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
396	4	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
397	5	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
398	6	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
399	7	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
400	8	mpi@r109-n60	SLAVE	Ray.r4140	1184258	r109-n60	8.15	8	20.2G	23.5G
401	1	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
402	2	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
403	3	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
404	4	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
405	5	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
406	6	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
407	7	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
408	8	mpi@r109-n61	SLAVE	Ray.r4140	1184258	r109-n61	8.00	8	20.6G	23.5G
409	1	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
410	2	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
411	3	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
412	4	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
413	5	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
414	6	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
415	7	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
416	8	mpi@r109-n62	SLAVE	Ray.r4140	1184258	r109-n62	8.00	8	20.2G	23.5G
417	1	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
418	2	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
419	3	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
420	4	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
421	5	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
422	6	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
423	7	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
424	8	mpi@r109-n63	SLAVE	Ray.r4140	1184258	r109-n63	8.00	8	20.2G	23.5G
425	1	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
426	2	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
427	3	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
428	4	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
429	5	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
430	6	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
431	7	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
432	8	mpi@r109-n65	SLAVE	Ray.r4140	1184258	r109-n65	8.00	8	20.3G	23.5G
433	1	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
434	2	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
435	3	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
436	4	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
437	5	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
438	6	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
439	7	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
440	8	mpi@r109-n66	SLAVE	Ray.r4140	1184258	r109-n66	8.00	8	20.2G	23.5G
441	1	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
442	2	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
443	3	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
444	4	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
445	5	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
446	6	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
447	7	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
448	8	mpi@r109-n7	SLAVE	Ray.r4140	1184258	r109-n7	8.14	8	20.6G	23.5G
449	1	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
450	2	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
451	3	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
452	4	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
453	5	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
454	6	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
455	7	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
456	8	mpi@r109-n8	SLAVE	Ray.r4140	1184258	r109-n8	8.14	8	20.7G	23.5G
457	1	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
458	2	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
459	3	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
460	4	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
461	5	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
462	6	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
463	7	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
464	8	mpi@r109-n9	SLAVE	Ray.r4140	1184258	r109-n9	8.14	8	20.6G	23.5G
465	1	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
466	2	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
467	3	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
468	4	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
469	5	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
470	6	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
471	7	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
472	8	mpi@r110-n11	SLAVE	Ray.r4140	1184258	r110-n11	8.04	8	20.5G	23.5G
473	1	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
474	2	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
475	3	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
476	4	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
477	5	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
478	6	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
479	7	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
480	8	mpi@r110-n14	SLAVE	Ray.r4140	1184258	r110-n14	8.15	8	20.2G	23.5G
481	1	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
482	2	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
483	3	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
484	4	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
485	5	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
486	6	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
487	7	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
488	8	mpi@r110-n15	SLAVE	Ray.r4140	1184258	r110-n15	8.00	8	20.1G	23.5G
489	1	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
490	2	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
491	3	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
492	4	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
493	5	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
494	6	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
495	7	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
496	8	mpi@r110-n19	SLAVE	Ray.r4140	1184258	r110-n19	8.15	8	20.2G	23.5G
497	1	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
498	2	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
499	3	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
500	4	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
501	5	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
502	6	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
503	7	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
504	8	mpi@r110-n2	SLAVE	Ray.r4140	1184258	r110-n2	8.15	8	20.2G	23.5G
505	1	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
506	2	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
507	3	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
508	4	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
509	5	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
510	6	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
511	7	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G
512	8	mpi@r110-n5	SLAVE	Ray.r4140	1184258	r110-n5	8.00	8	20.1G	23.5G

If you have read the table thoroughly, you presumably noticed the load of the compute node named r106-n61. It is 10.00 and it is off-limits. Reason's zombies. Yes, zombies. Previous jobs were not properly cleaned on completion and ghost processes plague some compute nodes, sometimes.

Séquençage haut-débit

2010-10-14T11:38:00.000-07:00

Sébastien Boisvert
Faculté de médecine de l'Université Laval
Centre de recherche en infectiologie de l'Université Laval

Le séquençage de l'ADN a longtemps été basé sur l'incorporation aléatoire de terminateurs qui inhibent l'ajout subséquent de nucléotides (Sanger et al., 1977). En se basant essentiellement sur une séparation par électrophorèse des molécules générées de longueur variable et sur une détection par fluorescence du nucléotide terminal de chaque molécule, la méthode de lecture de l'ADN du Dr. Sanger (Prix Nobel 1958 & 1980) fut automatisée par le laboratoire Hood (Smith et al., 1986) et commercialisée par Applied Biosystems. Le produit commercial principal de ces développements technologiques est le séquençeur à capillaires 3730 de Applied Biosystem.

Avec le développement de la biologie moléculaire, le début du projet du génome humain, et toutes les applications imaginables de la PCR (réaction en chaîne de la polymérase), beaucoup d'investissements financiers ont été fait pour accélérer le progrès technologique. En conséquence, des avancées extraordinaires dans le développement d'essais génomiques hautement parallèles, comme la PCR multiplexe, les puces à ADN d'Affymetrix et le séquençage haut-débit, ont été observées (Fan et al., 2006).

La compagnie 454 (achetée par Roche) a commercialisé un séquençeur parallèle d'ADN pouvant lire plusieurs centaines de milliers de gabarits -- sous la forme de séquences d'environ 230 nucléotides (Margulies et al., 2005). Maintenant, les séquenceurs 454 produisent des lectures d'environ 450 nucléotides. De plus, la méthode 454 n'utilise pas de terminateurs qui inhibent. Le séquenceur nécessite environ 8 heures pour compléter sa tâche.

Une autre compagnie appelée Solexa (achetée par Illumina) a commercialisé un séquenceur pouvant lire des dizaines de millions de gabarits d'ADN (Bentley et al., 2008). Les séquences générées étaient plus courtes, soit 36 nucléotides. Maintenant, un séquenceur Illumina produit des lectures pouvant aller jusqu'à 100 nucléotides. La technologie Illumina peut aussi lire les extrémités de molécules d'ADN plus longues pour ainsi générer des paires de séquences avec une distance qui les sépare qui est connue. Contrairement à la technologie 454, la technologie Illumina utilise des terminateurs réversibles. Le séquenceur nécessite environ une semaine pour compléter sa tâche.

Découvrir des variations génétiques

Une application évidente du séquençage à très haut débit est la découverte de mutations. Pour extraire les variations des séquences, il faut les aligner premièrement sur une séquence de référence (Figure 1).

Figure 1: Des séquences permettent de découvrir une variation génétique.

Plusieurs algorithmes ont été implémentés dans des logiciels libres, appelés aligneurs, afin de donner aux chercheurs les outils nécessaires pour répondre à des questions biologiques en utilisant le séquençage haut-débit. Par exemple, BWA -- pour Burrow-Wheeler Alignment Tool, est un logiciel dont l'algorithme utilise des routines avancées de compression (Li and Durbin, 2009a). Le format standard des analyses obtenues avec les aligneurs est appelé SAM (pour Sequence Alignment/Map) (Li. et al., 2009b).

Donner un sens aux séquences: algorithmes d'assemblage de novo

Mais parfois, un chercheur pourra avoir entre les mains des séquences d'un organisme vivant dont la séquence génomique est totalement inconnue. Dans une telle situation, il faut utiliser des logiciels implémentant des algorithmes d'assemblage de novo. Pour se faire, les bioinformaticiens utilisent la théorie des graphes, plus particulièrement les graphes de Bruijn (Pevzner et al., 2001). La théorie des graphes est utilisée depuis très longtemps pour procéder à l'analyse de séquences (Hutchinson et al., 1969). Avec cette approche, une séquence est convertie en graphe dirigée (Figure 2).

Figure 2: Une séquence courte d'ADN est transformée en graphe dirigé.

Dans la Figure 2, le graphe dirigé contient les sous-séquences de longueur 21 de la séquence, et les sous-séquences adjacentes dans la séquences sont liées par une flèche. En combinant la représentation en graphe de toutes les séquences, on obtient un graphe plus gros qui contient un chemin représentant la séquence génomique de laquelle proviennent les séquences (Boisvert et al., sous presse).

Références

Sanger, F. and Nicklen, S. and Coulson, A. R. (1977) DNA sequencing with chain-terminating inhibitors. Proc Natl Acad Sci U S A doi:10.1073/pnas.74.12.5463

Smith, L. M. et al. (1986) Fluorescence detection in automated DNA sequence analysis. Nature doi:10.1038/321674a0

Fan, J.-B. and Chee, M. S. and Gunderson, K. L. (2006) Highly parallel genomic assays. Nature Reviews Genetics doi:10.1038/nrg1901

Margulies, M. et al. (2005) Genome sequencing in microfabricated high-density picolitre reactors. Nature doi:10.1038/nature03959

Bentley, D. et al. (2008) Accurate whole human genome sequencing using reversible terminator chemistry. Nature doi:10.1038/nature07517

Li, H. and Durbin, R. (2009a) Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics doi:10.1093/bioinformatics/btp324

Li, H. et al. (2009b) The Sequence Alignment/Map format and SAMtools. Bioinformatics doi:10.1093/bioinformatics/btp352

Hutchinson, G. (1969) Evaluation of polymer sequence fragment data using graph theory. Bull Math Biophys doi:10.1007/BF02476636

Pevzner, P. A. and Tang, H. and Waterman, M. S. (2001) An Eulerian path approach to DNA fragment assembly. Proc Natl Acad Sci U S A doi:10.1073/pnas.171285098

Boisvert, S. and Laviolette, F. and Corbeil, J. (sous presse) Ray: simultaneous assembly of reads from a mix of high-throughput sequencing technologies. J Comput Biol.

(édité le 15 octobre 2010 pour des corrections mineures.)

Bible virology: pay to play?

2010-08-12T08:30:00.000-07:00

The authors of a paper about influenza in biblical times, which appeared online on 21 July 2010 in Virology Journal (operated by BioMed Central Ltd.), argue that "the woman with fever in the Bible is among one of the very early description of human influenza disease." Sébastien Boisvert reports.

Perhaps this is the most hilarious paper to have ever existed. And the bizarre event is even documented on Wikipedia.

On 1 August 2010, Paul A Gray, at Washington University School of Medicine in Saint-Louis, U.S.A., asked Virology Journal Editor-in-Chief Robert F Garry, at Tulane University School of Medicine in New Orleans, Louisiana, if "the Editor actually read this submission and decided that no peer review was necessary."

"I had my qualms about BioMedCentral journals and this only makes then stronger.", he added.

The Editor-in-Chief at Virology Journal then responded quickly stating clearly that "this article was read by an Editor and peer reviewed." But he hastely retracted the said paper the same day. Whether or not the paper went through peer review remains elusive.

This act was a deliberate attempt of open (read not anonymous) reviewing. The commenting system allowed a scientist to raise a red flag, and the publishing company responded accordingly.

Unfortunately, open review won't be around any time soon. "Despite enthusiasm for the concept, open peer review was not widely popular, either among authors or by scientists invited to comment." concluded Nature.

Robert F Garry did a poor job as a knowledge gatekeeper. Doubt persists as to whether or not it was peer-reviewed and assessed by an Associate Editor. In the affirmative, the required competence of the latter was lacking. Even a lay person would recognise the hilarious traits of the said work. And the title alone should turn on a warning light.

"I'm not a virologist, but if the standards for this journal are this lax, I wouldn't be wasting my time sending any of my papers there" wrote Tara C Smith, Assistant Professor of Epidemiology at College of Public Health at the University of Iowa.

"I'm not a virologist, but if the standards for this journal are this lax, I wouldn't be wasting my time sending any of my papers there." - Tara C Smith

Paul Zachary Myers -- a biologist and Associate Professor at the University of Minnesota, Morris -- wrote that his "favorite parts were the bits where the authors noted that Jesus did not take her temperature because the Fahrenheit scale wasn't invented until 1724."

Impending clash

According to Michael Nielsen, one of the pioneers of quantum computation, scientific publishing is about to be disrupted.

The null production cost of online publication is disrupting the industry. It is tearing it apart.

Institutions want highly-cited works (read high impact factor) to be associated with their names. And there is the Internet. Online-only publishing companies are a natural response to these. What we see is a plague against which no immune system can withhold.

Hilarity ensues.

Too few barriers are in place to inhibit such behaviours from open access publishers, one Paul A Gray called "pay to play".

To put it simply, it is perhaps a form of external evaluation. Surely the phenomena can be related to the concept of 'externality' in economics. It is the outsourcing of critical assessment of the every works of an institution's researchers. But what happens if that outsourced reviewing is performed by technological companies that truly lack science/medicine expertise.

Such an organisation receives a document, observes a delay (time adds prestige), collects currency, and uploads a PDF to a web server. The paper is then enhanced with at least a brand name -- the journal title backed with the publishing legacy of the company. While profiting of the financial wealth generated, such an enterprise can be perilous for the perpetuity of science, if not properly regulated.

Obviously, bombastic writings are not enough to make a good scientist, but surely they are convincing in the communication process with a for-profit publishing company.

Springer will soon produce more of these gems with their online-only SpringerOpen journals. As they put it, "SpringerOpen makes it easier than ever for authors to benefit from Springer's trusted brand."

Publishers such as Cold Spring Harbor Laboratory Press and Oxford University Press are desperately needed in these times of scientific publishing impending collapse.

The necessity of trust, that a received manuscript is surely scientifically sound, stirs too many concerns; too much is at stake.

Also on BioMed Central, but in a different paper, one can read that "the data have been released into pubic sequence repositories". Although this could be a typographical error, one can presume that peer-review was done hastely.

Citation:

Kam LE Hon, Pak C Ng and Ting F Leung. Influenza or not influenza: Analysis of a case of high fever that happened 2000 years ago in Biblical time. Virology Journal 2010, 7:169 doi:10.1186/1743-422X-7-169

20987491162

Partitions, genomes, and life sequences

2010-06-21T13:52:00.001-07:00

Tackling a problem by reducing it in smaller, more tractable parts, is the pathway that is followed to solve numerous, yet different, real-life problems. And divide-and-conquer can be rationally helpful to read through a genome's sequences of As, Ts, Cs, and Gs too. A method published in Genome Research facilitates reading genomes with partitioning. Sébastien Boisvert reports.

Reading life

Genomes contain the core code of life, or so we think. Recent technological advances are pushing for massively parallel acquisition of short digital sequences of DNA. However, these new sequencers -- machines that literally read DNA molecules -- can only decode about 30-150 digital letters at once of each DNA molecule among the many of them.

Accordingly, algorithms implemented as computer programs make sense from these sequence reads by assembling them in longer sequences -- akin to solving puzzles.

High-hanging fruit

Whole-genome shotgun short read sequencing and assembly are truly approachable for bacterial-sized genomes -- say 1-7 Mb. And otherwise all you get is thousands of unrelatable sequences. The lynchpin of genome biology is however to read larger genomes -- these of multi-cellular living organisms such as plants from the agricultural industry, and human genomes to harness knowledge and study thoroughly human health.

Genome partitions & life sequences

In the February 2010 issue of the scientific publication Genome Research, home of papers about -- obviously -- genome research, an international team of researchers from the National Human Genome Research Institute in the United States of America and the European Bioinformatics Institute in the United Kingdom proposed a novel approach to sequence larger genomes with current highly-parallel DNA sequencing instrumentation.

Low-flying fruits

The paper proposed a partitioning scheme to reduce the hardness of sequence assembly. The method's output is favorably compared to results obtainable by capillary-based methods, at a fraction of the cost and time. In the reported approach, the genome is cut at specific sites called restriction sites. Restriction enzymes recognize these and cut them.

According to the authors, these sites can be the workhorse for partitioning genomes.

The position of restriction sites depend on the nucleotide content of the genome to be sequenced. Thus, the spectrum of fragment lengths obtainable depends on the genome, too. The necessary a priori knowledge of these sites weakens, not to say nullifies, the authors' claims about the de novo nature of their published procedure.

The necessary a priori knowledge of these sites weakens, not to say nullifies, the authors' claims about the de novo nature of their published procedure.

The fly sites

In the paper, the authors suggested to use two specific restriction enzymes, and for each of them to select 4 different fragment lengths with gel electrophoresis.

The method was applied to the fruit fly Drosophila melanogaster 125-Mb genome, an overly studied life source code of an extensively characterized multi-cellular life form. As the scientists report, their fly genome assembly contained the target genetic blueprint contaminated by the blueprint of its microbiome.

De novo? not really.

Contrary to what the authors wrote -- that their method can decode the genome of yet unsequenced and uninvestigated living forms of life, it requires a thorough a priori knowledge of either the target genome or the one of its microbiome in order to fish the target genome, or to weed out the microbiome genome.

And the accurate choice of restriction enzymes surely requires a genome sequence.

To pair or not?

In 2008, the Illumina DNA sequencing technology could output paired sequences. The partition-and-sequence paper was received for peer-review on July 1st, 2009. One can read in that article that "the ability to generate accurate paired-end data was not available to us when we initiated this project." The lack of pairs is then solely based on the definition of the word "accurate".

Genome constructs to nonsense constructs

On page 255, one can confusingly read that "The RR libraries were sequenced individually using Velvet."

Reading DNA molecules with an intangible genetic-analysis software is of course a would-be hit, not the century's breakthrough. An inadequate wording or an editing error presumably explains it.

Fair enough or not enough?

The method is not readily-applicable for organisms whose genome are unknown, because otherwise the resulting strings would be a undesirable mix. Moreover, the knowledge of the genome sequence is a prelude to the adequate choice of restriction enzymes.

Genome partitioning is a good debut, but the method reported is inadequate and overfits the fly.

The method is not conceptually so different from existing DNA capture-based enrichment methods that aim at partitioning genomes: they both need genome sequences.

Reference

Genome Res. 2010. 20: 249-256 doi:10.1101/gr.097956.109

Planification de mon doctorat

2010-04-16T09:03:00.000-07:00

Programme d'études: http://capsuleweb.ulaval.ca/pls/etprod7/y_bwckprog.p_afficher_fiche?p_session=200909&p_code_prog=D-PEN&p_code_majr=PEN&p_code_camp=

Session 1/11 -- Janvier 2010 à Avril 2010

PHS-8801 Activité de recherche - thèse 1

Session 2/11 -- Mai 2010 à Août 2010

PHS-8802 Activité de recherche - thèse 2

Session 3/11 -- Septembre 2010 à Décembre 2010

BIF-7900 Bio-informatique I A+
PHS-8803 Activité de recherche - thèse 3

Session 4/11 -- Janvier 2011 à Avril 2011

PHS-8000 Examen de doctorat (écrit: 1 mars 2011, oral: 1 avril 2011)
PHS-8804 Activité de recherche - thèse 4

Session 5/11 -- Mai 2011 à Août 2011

PHS-8805 Activité de recherche - thèse 5

Session 6/11 -- Septembre 2011 à Décembre 2011

PHS-8001 Séminaire de doctorat
PHS-8806 Activité de recherche - thèse 6

Session 7/11 -- Janvier 2012 à Avril 2012

BMO-7007 Analyse critique de publications
PHS-8807 Activité de recherche - thèse 7

Session 8/11 -- Mai 2012 à Août 2012

PHS-8808 Activité de recherche - thèse 8

Session 9/11 -- Septembre 2012 à Décembre 2012

Rédaction

Session 10/11 -- Janvier 2013 à Avril 2013

Rédaction

Session 11/11 -- Mai 2013 à Août 2013

Rédaction et dépôt de thèse

Doctoral Award

2010-04-15T21:07:00.000-07:00

Detailed Information

Project Information
Project Title:	Assembly of reads from a mix of high-throughput sequencing technologies
Period:	from: 2010-09-01 to: 2013-08-31
Principal Investigator(s):	BOISVERT, Sébastien
Co-Investigators:
Supervisors:	CORBEIL, Jacques ; MARCHAND, Mario
Previous Investigators/Supervisors:
Institution:	Université Laval
Department:
Program:	Doctoral Award - Frederick Banting and Charles Best Canada Graduate Scholarships
Assigned PRC:	DRA
Primary Institute:	Genetics
Primary Theme:	Biomedical
Keywords:	BIOINFORMATICS, BIOSTATISTICS, BIOSTATISTIQUES, DE NOVO SEQUENCING, GENOME ASSEMBLY, GENOME VARIATION, GENOMICS, PROTEOMICS, AND BIOINFORMATICS, GÉNOMIQUE, PROTÉOMIQUE ET BIOINFORMATIQUE, MUTATION DISCOVERY, RESEQUENCING
Abstract:	The treatment of diseases costs each Canadian a large amount of money. There is a genetic basis behind resistance to antibiotics and various genetics diseases. Genetic background accounts for predisposition to diseases, and the knowledge of the latter can promote better lifestyle. Sequencing technologies allow physicians to diagnose precisely the cause of some genetics disease, and thus explain patients the very cause of their sickness. When a resistant strain hits hard on a health care centre, the rapid identification of the pathogen agent is essential to avoid unnecessary lost of human lives. Genetic tests and pathogen detection kits are based on nucleic acid technologies, such as high-density micro-arrays and sequencing technologies. In the case of genetic diseases, these technologies allow the pinpointing of malfunctioning proteins, and the latter are addressed with tailored drugs, if any. There is a great need in Canada for people that can analyse the data produced by these DNA technologies -- we call them bioinformaticians. Owing to the complexity of the human genome and to the evolution of the associated technological medicine, new powerful tools are necessary to analyze the sheer volume of data. The knowledge of mutations that set the predisposition to known diseases, such as Alzheimer's disease, surely improves on the quality of life. The rapid detection and characterisation of disease agents reduces the chance of outbreak, and improves population health. In the next decade, sequencing technologies will allow the study of human genome variation through the sequencing of a thousand human genomes, offering a deep catalog of human genetic variation (http://1000genomes.org/). During my doctorate, I will develop tools to answer these important health questions in genomics.

Funding Information
Fiscal Year	Amount
2010-11	$35,000

Source: http://webapps.cihr-irsc.gc.ca/funding/

Master's Award

2010-04-15T21:06:00.000-07:00

Detailed Information

Project Information
Project Title:	Kernel methods for the annotation of trypanosomatid genomes.
Period:	from: 2009-09-01 to: 2010-08-31
Principal Investigator(s):	BOISVERT, Sébastien
Co-Investigators:
Supervisors:	CORBEIL, Jacques ; MARCHAND, Mario
Previous Investigators/Supervisors:
Institution:	Centre hospitalier de l'Université Laval (Québec)
Department:	Molecular Medicine
Program:	Frederick Banting and Charles Best Canada Graduate Scholarships - Master's Award
Assigned PRC:	CGA
Primary Institute:	Genetics
Primary Theme:	Biomedical
Keywords:	BIOINFORMATICS, COMPARATIVE GENETICS, GENETICS, GENOME ANNOTATION, GENOMICS, GENOMICS, PROTEOMICS, AND BIOINFORMATICS, GÉNOMIQUE, PROTÉOMIQUE ET BIOINFORMATIQUE, GÉNÉTIQUE, HIV, INFECTIEUSES ET PARASITAIRES, INFECTIOUS AND PARASITIC, INFECTIOUS MICROORGANISMS, KERNEL METHODS, LEISHMANIA, MACHINE LEARNING, TRYPANOSOMATIDS
Abstract:	Not Applicable

Funding Information
Fiscal Year	Amount
2009-10	$17,500

Source: http://webapps.cihr-irsc.gc.ca/funding/

SHREC mines errors

2009-10-11T08:32:00.000-07:00

SHREC: a short-read error correction method
Bioinformatics doi:10.1093/bioinformatics/btp379 (2009)

In their paper, the authors stress on the importance of read correction in sequencing applications -- such as resequencing and de novo assembly. They mention that error correction techniques for Sanger reads are outdated, and that novelty is required. The authors say that assemblers are better when there're no errors -- that they work well with error-corrected reads. This is incorrect, as both the resequencing and de novo assembly applications, allow consensus calling. The authors write that the Euler assembly program is 'established', meanwhile, nobody uses it. Over the Sanger reign, only a very few software were introduced for error correction in reads. Accordingly, the authors only cite two such works.

They compare SHREC with the error-correction component of EULER-SR and ALLPATHS -- two assemblers -- on simulated and real data. The paper is fun to read, and the described work is sound and easy to grasp. The author are careful and scientific, they define every terms they use. Their method outperforms the others, and the approach is easy to use on a computer. The authors generated random errors, and utilized Illumina reads -- which also include solely randomly-located errors. However, the use of coverage depth only addresses minor errors, and it ignores correlated errors, such as the 454's insertions and deletions in homopolymers.

The novelty of the paper is at the rendez-vous, the work is well-motivated, the software is open-source, and benchmarks are adequately reported.

Polish researchers elucidate genome assembly

2009-10-10T14:43:00.000-07:00

Whole genome assembly from 454 sequencing output via modified DNA graph concept
Computational Biology and Chemistry doi:10.1016/j.compbiolchem.2009.04.005 (2009)

The human genome project was a scientific success which allowed bioinformatics to grow. During this project, only Sanger sequencing was in action. Recently, with pyrosequencing, however, new platforms are emerging, and they provide much more data at lower cost, in a few hours. This data storm has prompted the need for novel assembly algorithms.

The authors provide a new computational framework for genome assembly -- SR-ASM (Short Reads ASseMbly). They utilize the 'recently available' Roche/454 technology, released in 2005. They evaluate their tool against Velvet and Newbler. Velvet is designed solely for Illumina (the authors say it runs on 454), whereas Newbler is sold along with the 454 sequencer from Roche. The authors say Newbler can not load fasta files, but it can. With this argument, they avoided the need to compare their tool with Roche's software -- the mighty Newbler. Roche/454 is a commercial success, mainly because of Newbler. The authors write that Phrap is not 454-aware; this is incorrect. The 454 sequencer takes 7 h to read DNA, Newbler assembles reads in about 15 minutes. Meanwhile, SR-ASM runs in only 80 hours.

Tests were done on the Prochlorococcus marinus 1.84 Mbp genome, and a 11717 nt segment of human chromosome 15. The authors write that a lack of coverage in 454 experiments causes assembly gaps. In principle, this is incorrect -- repeated elements are most likely to be responsible because novel sequencing technologies offer large depth and breadth of coverage. In their paper, tables are incomprehensible, and they don't capture the true value of assemblies. The authors only check for contig lengths, they don't assess the assembly errors.

In summary, the paper is hard to read, not scientific, and the novelty is very weak, if not inexistant.

Beware of bioinformatics

2009-10-06T13:49:00.000-07:00

The advent of computing in our world has made research endeavours easier. The whole field of bioinformatics is built on the mindset that once you have data, regardless of its quality, you can go on and publish the data-analysis recipe and associated observations. Why do computational biologists feel they have to tell everyone that their sequence contains specific sub-sequences? Even more alarming, unless you are publishing in a good journal -- like Nature -- a bioinformatic paper is very unlikely to undergo extensive copy-editing, and thus, will presumably be super-boring to read, unless you are an outstanding writer. Suppose that a person prepares bread with his own novel recipe (yeah, right -- a novel bread recipe). He mixes the ingredients rightly, then he put the mix in the oven, sets it up, waits, and gets his result. This result is void of any scientific value, just as most of the creepy crap published in the bioinformatics sphere is void of discoveries. One could tell that this lack of discoveries is brought by the methodological aspects of bioinformatics -- that bioinformatics in essence is about how you discover things, not about discovering them. Let me consider the persistent problem of short-read alignment, which is pervasive in resequencing adventures. Considering that a genome sequence is very lengthy, and that those small chunks of data obtained are very short, but long enough to allow unique placement when they occur in unique genome segments, then even a child could design a novel algorithm. People are smashing themselves in walls to publish their own programs that align reads onto genomes. Whoa, reads!, genomes!, my nature, this is complex science. Not it's not. Overall, the open access movement has sparked many publishing start-ups, whose commercial activities lay in the process of making available online manuscripts that they receive, along with about a thousand boxes. Sure this is a winning business model, given that 'genomics' and 'computation' are power words.

Below, I attached a carefully-prepared list of peer-reviewed journals in bioinformatics. Go check them out!

Algorithms in molecular biology
Bioinformatics
Briefings in Bioinformatics
BMC Bioinformatics
BMC Genomics
Computational Biology and Chemistry
Journal of Bioinformatics and Computational Biology
Journal of Computational Biology
PLoS Computational Biology
The Open Bioinformatics Journal

(a more detailed list is available here)

Edits: 2010-06-22 (typos)

Updates on my assembler software

2009-02-02T20:38:00.000-08:00

I have been working very hard since December 10th on a novel algorithm for fragments assembly. I already registered a sourceforge open source project. I started with some of the ideas of Pevzner et al. 2001, but I added several enhancements, and I modeled the problems with equations. My software is compatible with the amos specification and will be released on sourceforge upon acceptance in a journal. We are planning to submit our work to PNAS. We collected public data sets from the Short Read Archive to assess the performance of our software.

The Distant Segments Kernel: a tutorial

2008-12-07T10:11:00.000-08:00

This post is a tutorial on how to use the distant segments kernel. String kernels were recently introduced as a more precise way to perform pairwise comparisons. First, I will define the concept of kernels. A kernel is a function that takes two objects in an input space and multiply them by mapping them to a vectorial feature space. A kernel associates a real number to a pair of instances. String kernels are a particular case of kernels. The input space of string kernels is the set of strings. Recall that strings are sequences generated with a given alphabet. The distant segments kernel is a string kernel. For the distant segments kernel, the feature vector associated to a string is the distribution of its distant segments. See this paper for more details.

In this tutorial, command lines are shown in red. First, download the source code.

seb@ubuntu:~$ wget http://boisvert.info/software/PermutationDSKernel.cpp

This software performs the kernel matrix computation of a set of strings. Now, compile the c++ file with g++.

seb@ubuntu:~$ g++ PermutationDSKernel.cpp -O4 -Wall -o DSK

Executing the program without providing parameters will output the usage.

seb@ubuntu:~$ ./DSK
Usage:
DSkernel l sequenceFile delta_m theta_m out

Basically, sequenceFile is a file containing sequences. In this file, you must put one sequence per line. Note that this file is not a fasta file.

In this tutorial, I will use the file sequences.txt. You can copy the content below in sequences.txt on your computer. The sequences.txt file contains protein sequences from the SCOP database.

seb@ubuntu:~$ cat sequences.txt
svydaaaqltadvkkdlrdswkvigsdkkgngvalmttlfadnqetigyfkrlgnvsqgmandklrghsitlmyalqnfidqldnpddlvcvvekfavnhitrkisaaefgkingpikkvlasknfgdkyanawaklvavvqaal
lsaaqkdnvksswakasaawgtagpeffmalfdahddvfakfsglfsgaakgtvkntpemaaqaqsfkglvsnwvdnldnagalegqcktfaanhkargisagqleaafkvlagfmksyggdegawtavagalmgmirpdm
glsaaqrqviaatwkdiagadngagvgkkclikflsahpqmaavfgfsgasdpgvaalgakvlaqigvavshlgdegkmvaqmkavgvrhkgygnkhikaqyfeplgasllsamehriggkmnaaakdawaaayadisgalisglqs
vlsegewqlvlhvwakveadvaghgqdilirlfkshpetlekfdrfkhlkteaemkasedlkkhgvtvltalgailkkkghheaelkplaqshatkhkipikylefiseaiihvlhsrhpgdfgadaqgamnkalelfrkdiaakykelgy
xslsaaeadlagkswapvfanknangldflvalfekfpdsanffadfkgksvadikaspklrdvssriftrlnefvnnaanagkmsamlsqfakehvgfgvgsaqfenvrsmfpgfvasvaappagadaawtklfgliidalkaaga
lsadqistvqasfdkvkgdpvgilyavfkadpsimakftqfagkdlesikgtapfethanrivgffskiigelpnieadvntfvashkprgvthdqlnnfragfvsymkahtdfagaeaawgatldtffgmifskm
seb@ubuntu:~$ wc -l sequences.txt
6 sequences.txt

To perform a pairwise comparison, one can use string kernels. Note that pairwise comparisons are usually done with alignment software. Instead of relying on an alignment to perform this task, here we will utilize the computer program that we just compiled from its source code.

seb@ubuntu:~$ ./DSK 6 sequences.txt 1000 3 KernelMatrix-DSK-1000-3.txt

This performed the computation for the 6 sequences in sequences.txt with the distant segments kernel. Hyperparameters deltaM and theteM were set respectively to 1000 and 3. See the paper for a clear explanation of the role of those hyperparameters in the distant segments kernel. The last argument that we provided is simply the output file.

The output file contains a square matrix. For each pair of sequences, it contains the image of the distant segments kernel. Let us look at the content of this file.

seb@ubuntu:~$ cat KernelMatrix-DSK-1000-3.txt
x 0 1 2 3 4 5
0 95925 5707 6163 5114 6218 4130
1 5707 93627 9197 5396 9088 5900
2 6163 9197 103266 6711 8617 5547
3 5114 5396 6711 105790 5833 4359
4 6218 9088 8617 5833 101886 6256
5 4130 5900 5547 4359 6256 83565

As we can see, a diagonal dominance occurs and inner products are large. The distant segments kernel has broad applicability. One of the applications is to use the distant segments kernel as the similarity operator of the support vector machine. svmlight, an implementation of the SVM algorithm, provides ways to plug in any kernel. You can download this package to use a precomputed matrix with svmlight. Note that you need to download svmlight separately here. Note also that svmlight is free for academic use.

For real-world bioinformatics application, the kernel matrix needs to be normalized. One of the reasons is that proteins share subsequences, but have a varying length. One way to normalize the matrix is to transform all associated vectors to vectors of unit norm. Here, in the last part of this tutorial, I show how to normalize the kernel matrix associated to the distant segments kernel. Let us compile the computer program Normalize.

seb@ubuntu:~$ wget http://genome.ulaval.ca/dav/boiseb01/pub/software/svmoptimize-v1.zip
seb@ubuntu:~$ unzip svmoptimize-v1.zip
seb@ubuntu:~$ gcc -O4 -Wall svmoptimize-v1/Normalize.c -o svmoptimize-v1/Normalize -lm

Again, executing the program without arguments will show the correct usage.

seb@ubuntu:~$ ./svmoptimize-v1/Normalize
usage: program matrix n
k'(x,y) <- k(x,y)/sqrt(k(x,x) k(y,y))

The normalized matrix is generated and written to KernelMatrix-DSK-1000-3.txt.1

seb@ubuntu:~$ ./svmoptimize-v1/Normalize KernelMatrix-DSK-1000-3.txt 6 > KernelMatrix-DSK-1000-3.txt.1
seb@ubuntu:~$ cat KernelMatrix-DSK-1000-3.txt.1
x 0 1 2 3 4 5
0 1.000000 0.060220 0.061922 0.050766 0.062897 0.046129
1 0.060220 1.000000 0.093533 0.054219 0.093049 0.066702
2 0.061922 0.093533 1.000000 0.064208 0.084008 0.059713
3 0.050766 0.054219 0.064208 1.000000 0.056184 0.046361
4 0.062897 0.093049 0.084008 0.056184 1.000000 0.067800
5 0.046129 0.066702 0.059713 0.046361 0.067800 1.000000

Using a normalized matrix keeps the same angle between proteins, but sets the norm of proteins to 1. Presumably, the angle between proteins is more important than the norm of proteins. This normalized matrix is suitable for supervised learning with the SVM, if labels are available. This concludes this tutorial.

References:

Sebastien Boisvert, Mario Marchand, Francois Laviolette, and Jacques Corbeil. Hiv-1 coreceptor usage prediction without multiple alignments: an application of string kernels. Retrovirology, 5(1):110, Dec 2008. [ bib | DOI | http ]

My research activities

2008-12-02T20:41:00.001-08:00

I started my master's degree in September. I also have a new web site. If everything goes as planned,I should start a PhD with professor Jacques Corbeil and professor Mario Marchand at the Université Laval by next year. So far, in my two first internships I focused on microarray for gene expression (Ubeda et al., Rochette et al.). Last year, in my third internship, I also had the chance to develop a new method for the prediction of HIV coreceptor usage with Mario Marchand, François Laviolette and Jacques Corbeil. Our manuscript was recently accepted for publication in the journal Retrovirology (to appear). Since existing methods for this particular task all need each V3 loop (a component of HIV) to be aligned, we saw a good opportunity to apply string kernels. Recall that a kernel is a similary measure and that it maps objects to a feature space to perform a dot product. Recall also that string kernels are the family of kernels whose input space is the set of strings. Alignments can be the cause of many problems. Just to name one, alignments break the i.i.d. hypothesis (each example are identically and independantly distributed according to an unknown, but constant distribution). In particular, cross-validation can not be performed on a set of sequences that were aligned to each other because the i.i.d. hypothesis is broken. Consequently, we developped the distant segments kernel (the source of inspiration for naming my blog), a kernel with a very large feature space. Its feature space is constructed by counting the distances between pairs of segments inside the primary structure of a protein. I also tested this kernel on the SCOP data set. With SVMlight (C=10) and the distant segments kernel (theta_m=3, delta_m=1000), I obtained very competitive results (i.g. ROC AUC). The mean AUC is 0.9265 for the 54 families. This is impressive because I did not applied any twisted changes to the diagonal of the kernel matrix. The kernel matrix was generated and feature vectors were implicitly normalized to unit norm. Also, positive examples were duplicated in the training sets until a balance (i.g. 1:1) is achieved. However, I don't think that the results on this data set are significant, for any algorithm, because the training and test sets are not build from the same distribution (see this paper). During the next year, I want to assign biological functions to some of the trypasomatid proteins with unknown function. To tackle this task, I will employ presumably kernel methods. However, the best approach has not yet been determined by our marvelous team. This problem can be interpreted as a ranking problem, as a supervised learning problem (see this data set), or even in a semi-supervised learning problem. Currently, I am working on the assembly and annotation of the genome of Leishmania tarentolae. Also, I maintain an online bibliography.