A catalog of IBM Blue Gene/Q errors (for science)

-
Once in a while, I get to run things on a IBM Blue Gene/Q. And when that happens, some of my jobs always crash with random errors.

For science, here are some of them.

Update 2013-02-27 with MPI I/O: 

 This requires fcntl(2) to be implemented. As of 8/25/2011 it is not. Generic MPICH Message: File locking failed in ADIOI_Set_lock(fd 4,cmd F_SETLKW/E,type F_WRLCK/1,whence 0) with return value FFFFFFFF and errno 23.
- If the file system is NFS, you need to use NFS version 3, ensure that the lockd daemon is running on all the machines, and mount the directory with the 'noac' option (no attribute caching).
- If the file system is LUSTRE, ensure that the directory is mounted with the 'flock' option.
ADIOI_Set_lock:: Resource deadlock avoided
ADIOI_Set_lock:offset 25625204815, length 6282003
Abort(1) on node 2501 (rank 2501 in comm 1140850688): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2501

SRA056234-Picea-glauca-2013-02-12-1

2013-02-13 04:56:26.627 (FATAL) [0x40001138a50] :1555:ibm.runjob.client.Job: terminated due to: killing the job timed out
2013-02-13 04:56:26.628 (FATAL) [0x40001138a50] :1555:ibm.runjob.client.Job: abnormal termination by signal 35 from rank 2712 due to RAS event with record ID 279083. END_JOB control action heartbeat timed out after 60 seconds
2013-02-13 04:56:26.628 (FATAL) [0x40001138a50] :1555:ibm.runjob.client.Job: 937 RAS events
2013-02-13 04:56:26.628 (FATAL) [0x40001138a50] :1555:ibm.runjob.client.Job: most recent RAS event text: CFAM Machine Check. Message=REASON: Core4 failed (uncorrectable error).  DETAILS: CFAM_Status=0xc0000000, MachineCheck [Core 4 Chiplet chkstp reg=0x8400000000000000: , Summary bit for xfir_lt, Chkstp from FIR1 [Core 4 PCB FIR1=0x0000000020000000: , A2-L2 UE]]!, DrillDown=CFAM_Status=0xc0000000, MachineCheck [Core 4 Chiplet chkstp reg=0x8400000000000000: , Summary bit for xfir_lt, Chkstp from FIR1 [Core 4 PCB FIR1=0x0000000020000000: , A2-L2 UE]]!


SRA056234-Picea-glauca-2012-12-22-13

2012-12-22 21:40:47.545 (FATAL) [0x40000ee8a50] :23842:ibm.runjob.client.Job: could not start job: block is unavailable due to a previous failure
2012-12-22 21:40:47.546 (FATAL) [0x40000ee8a50] :23842:ibm.runjob.client.Job: node R00-M0-N00-J00 is not available: Software Failure


SRA056234-Picea-glauca-2013-01-18-15

2013-01-19 13:49:22.860 (FATAL) [0x40001138a50] :1474:ibm.runjob.client.Job: terminated due to: killing the job timed out
2013-01-19 13:49:22.861 (FATAL) [0x40001138a50] :1474:ibm.runjob.client.Job: abnormal termination by signal 35 from rank 3345 due to RAS event with record ID 259250. END_JOB control action heartbeat timed out after 120 seconds
2013-01-19 13:49:22.861 (FATAL) [0x40001138a50] :1474:ibm.runjob.client.Job: 189 RAS events
2013-01-19 13:49:22.861 (FATAL) [0x40001138a50] :1474:ibm.runjob.client.Job: most recent RAS event text: A BQL double bit error threshold was exceeded for Switch 2 Group EVEN and ODD


SRA056234-Picea-glauca-2013-02-10-1

2013-02-11 16:10:37.815 (WARN ) [0x40001138a50] :ibm.runjob.LogSignalInfo: received signal 15
2013-02-11 16:10:37.815 (WARN ) [0x40001138a50] :ibm.runjob.LogSignalInfo: signal sent from USER
2013-02-11 16:10:37.815 (WARN ) [0x40001138a50] :ibm.runjob.LogSignalInfo: sent from pid 12709
2013-02-11 16:10:37.816 (WARN ) [0x40001138a50] :ibm.runjob.LogSignalInfo: could not read /proc/12709/exe
2013-02-11 16:10:37.816 (WARN ) [0x40001138a50] :ibm.runjob.LogSignalInfo: Permission denied
2013-02-11 16:10:37.817 (WARN ) [0x40001138a50] :ibm.runjob.LogSignalInfo: sent from uid 0 (root)
2013-02-11 16:10:42.677 (WARN ) [0x40001138a50] :1553:ibm.runjob.client.Job: terminated by signal 9
2013-02-11 16:10:42.677 (WARN ) [0x40001138a50] :1553:ibm.runjob.client.Job: abnormal termination by signal 9 from rank 10513
2013-02-11 16:10:42.677 (WARN ) [0x40001138a50] :1553:ibm.runjob.client.Job: 139 RAS events
2013-02-11 16:10:42.677 (WARN ) [0x40001138a50] :1553:ibm.runjob.client.Job: most recent RAS event text: DDR Correctable Error Summary : count=1 MCFIR error status:  [MEMORY_CE] This bit is set when a memory CE is detected on a non-maintenance memory read op;


Comments

Rob said…
To avoid the MPI-IO locking error, try prefixing your file name with bglockless:

Friday, April 25, 2014 at 3:06:00 PM EDT
Blogger said…
Did you know that that you can make cash by locking selected sections of your blog / site?
Simply join AdWorkMedia and embed their Content Locking widget.
Tuesday, October 11, 2016 at 2:30:00 PM EDT
Post a Comment

Popular posts from this blog

A survey of the burgeoning industry of cloud genomics

-
Image
I like lists. Here is a list of companies in the cloud genomics industry. Table 1: Companies acting in the industry of cloud genomics. The list is in no particular order. Company Products People Founded Link GenoSpace, LLC - GenoSpace platform - John Quackenbush (CEO) 2011 https://www.genospace.com/ DNAnexus, Inc. - DNAnexus Platform - DNAnexus Platform SDK (a.k.a. dx-toolkit) - Serafim Batzoglou (co-founder) - Andreas Sundquist (co-founder and CTO) - a bunch of other famous people in the field 2009 https://dnanexus.com/ Seven Bridges Genomics, Inc. - IGOR - IGOR Python SDK - Igor Bogicevic (CTO) 2009 https://www.sbgenomics.com/ BGI - EasyGenomics - Xing Xu (Senior Product Manager) Introduced in 2012 https://www.easygenomics.com/ Illumina, Inc. - BaseSpace™ - BaseSpace API - Alex Dickinson (SVP of Cloud Genomics) Introduced in 2011 https://basespace.illumina.com/ Era7 Information Technologies SLU - BG7...
Read more

Generating neural machine instructions for multi-head attention

-
  An animal can do training and inference every day of its existence until the day of its death. A forward pass is all you need.   Here is the current model configuration that I use for multi-head attention:         let sequence_length = 6;         let vocab_size = 20;         let n_embd = 64;         let num_heads = 8; You can see the full model here: https://github.com/sebhtml/novigrad/blob/8df6192829edd9cda3a221e12affa770fbda2a1b/src/models/mega_man_attention.rs The struct uses Embedding, MultiHeadAttention, Linear, Softmax: struct MegaManAttentionModel {     input_shape: Vec<usize>,     output_shape: Vec<usize>,     vocab_size: usize,     sequence_length: usize,     embedding: Embedding,     multi_head_attention: MultiHeadAttention, ...
Read more

Adding ZVOL VIRTIO disks to a guest running on a host with the FreeBSD BHYVE hypervisor

-
On atlantis.domain (my FreeBSD BHYVE hypervisor host), I use these commands: Install FreeBSD on a 10G disk. vm create -s 10G guestname   vm install FreeBSD-12.1-RELEASE-amd64-memstick.img vm console guestname Add 8 30GB zvol-based disks vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname vm add -d disk -t zvol -s 30G guestname You will need to do vm poweroff guestname vm start guestname because the guest does not seem to see the 8 30GB without power cycling. (rebooting won't work) *** If you want the boot disk (the one create by "vm create") to be a zvol, you must edit the template: root@atlantis:~ # zfs snapshot tank/vm-bhyve@2020-06-11-2145 root@atlantis:~ # mount -t zfs tank/vm-bhyve@2020-06-11-2145 /tank/vm-bhyve@2020-06-11-2145 r...
Read more