ORA-15064 错误解决

整理一下前段时间遇到的ORA-15064错误。有一套测试环境是11gR2的rac使用ASM，应用在测试时发现一个节点宕机了，检查了一下alert日志如下错误：Mon Jun 04 23:27:32 2012NOTE: ASMB terminating
Errors in file /soft/Oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
Errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
ASMB （ospid: 13828564）: terminating the instance due to error 15064
Termination issued to instance processes. Waiting for the processes to exit
Mon Jun 04 23:27:43 2012
Instance termination failed to kill one or more processes
Instance terminated by ASMB, pid = 13828564

通过错误可知，改错无与ASM有关，应该是ASM意外关闭才导致的Instance关闭。
错误的trace文件内容如下：
[pt1rac2@root]# cat /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trcTrace file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /soft/oracle/product/11.2.0.2/dbh
System name: AIX
Node name: pt1rac2
Release: 1
Version: 6
Machine: 00F770E14C00
Instance name: pt1db2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 27
Unix process pid: 13828564, image: oracle@pt1rac2 （ASMB）

*** 2012-06-04 16:19:14.483
*** SESSION ID:（2998.1） 2012-06-04 16:19:14.483
*** CLIENT ID:（） 2012-06-04 16:19:14.483
*** SERVICE NAME:（） 2012-06-04 16:19:14.483
*** MODULE NAME:（） 2012-06-04 16:19:14.483
*** ACTION NAME:（） 2012-06-04 16:19:14.483

NOTE: initiating MARK startup

*** 2012-06-04 16:19:18.290
Mode of disk 0 （DATA_0000） changing from 0x7 to 0x7

*** 2012-06-04 23:07:12.286
Mode of disk 0 （DATA_0000） changing from 0x7 to 0x7

*** 2012-06-04 23:27:32.492
NOTE: ASMB terminating
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
error 15064 detected in background process
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts（）+644<-kjzdssdmp（）+444<-kjzduptcctx（）+272<-kjzdicrshnfy（）+96<-ksuitm（）+1284<-ksbrdp（）+4508<-opirip（）+1216<-opidrv（）+1156<-sou2o（）+192<-opimai_real（）+308<-ssthrdmain（）+340<-main（）+216<-__start（）+112
----- End of Abridged Call Stack Trace -----

*** 2012-06-04 23:27:32.854
ASMB （ospid: 13828564）: terminating the instance due to error 15064

*** 2012-06-04 23:27:43.387
Instance termination failed to kill one or more processes

可知的确是由于ASM意外关闭才导致的instance关闭，ASM意外关闭的原因继续查。准备生产一下awr报告看一下，结果发现由于数据关闭了，没有awr可看。一下数据说明数据库被重启过几次，且经过与应用人员确认，这些重启的时间点他们都在做压力测试，因此初步怀疑是和性能有关，继续查。
Snap
Instance DB Name Snap Id Snap Started Level
------------ ------------ --------- ------------------ -----
pt1db2 PT1DB 503 02 Jun 2012 16:00 1
504 02 Jun 2012 17:00 1

505 04 Jun 2012 09:43 1

506 04 Jun 2012 11:00 1
507 04 Jun 2012 12:00 1
508 04 Jun 2012 13:00 1
509 04 Jun 2012 14:00 1
510 04 Jun 2012 15:00 1
511 04 Jun 2012 16:00 1
512 04 Jun 2012 17:00 1
513 04 Jun 2012 18:00 1
514 04 Jun 2012 19:00 1
515 04 Jun 2012 20:00 1
516 04 Jun 2012 21:00 1
517 04 Jun 2012 22:00 1
518 04 Jun 2012 23:00 1

522 05 Jun 2012 01:20 1
523 05 Jun 2012 01:30 1
524 05 Jun 2012 01:45 1

先查看ocssd.log日志
2012-06-04 23:26:02.274: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 113598 msecs
2012-06-04 23:26:03.126: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.126: [ CSSD][1029]clssgmAllocProc: （11328bcd0） allocated
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11328bcd0 - 0,1,2,3,4
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2ea97f） proc（11328bcd0） pid（15335558） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.274: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:03.274: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:03.309: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.309: [ CSSD][1029]clssgmAllocProc: （1132d30d0） allocated
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132d30d0 - 0,1,2,3,4
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2ea9fa） proc（1132d30d0） pid（9306434/9306434） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.315: [ CSSD][1029]clssscSelect: cookie accept request 1132d30d0
2012-06-04 23:26:03.315: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132d30d0
2012-06-04 23:26:03.316: [ CSSD][1029]clssgmRegisterClient: proc（57/1132d30d0）, client（1/11313ef70）
2012-06-04 23:26:03.316: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDeadProc: proc 112f32d30
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDestroyProc: cleaning up proc（112f32d30） con（2ea049） skgpid ospid 35520704 with 0 clients, refcount 0
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea049
2012-06-04 23:26:29.675: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:29.675: [ CSSD][1029]clssgmAllocProc: （113242130） allocated
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113242130 - 0,1,2,3,4
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eadef） proc（113242130） pid（10879472） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.226: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.226: [ CSSD][1029]clssgmAllocProc: （1132a3390） allocated
2012-06-04 23:26:30.227: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132a3390 - 0,1,2,3,4
2012-06-04 23:26:30.227: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eae64） proc（1132a3390） pid（35520706） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.228: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.542: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.542: [ CSSD][1029]clssgmAllocProc: （1132ad030） allocated
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132ad030 - 0,1,2,3,4
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eaede） proc（1132ad030） pid（8585490/8585490） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.553: [ CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.553: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.553: [ CSSD][1029]clssgmRegisterClient: proc（63/1132ad030）, client（1/1132ad370）
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmQueueShare: （1132ad510） target global grock DG_DATA member 1 type 1 queued from client （1132ad370）, global grock DG_DATA, refcount 15
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 15
2012-06-04 23:26:30.557: [ CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.557: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.557: [ CSSD][1029]clssgmRegisterClient: proc（63/1132ad030）, client（2/1132a3d90）
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmQueueShare: （1132ad910） target global grock DB+ASM member 1 type 1 queued from client （1132a3d90）, global grock DB+ASM, refcount 17
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 17
2012-06-04 23:26:30.636: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.637: [ CSSD][1029]clssgmAllocProc: （11333ba70） allocated
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11333ba70 - 0,1,2,3,4
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eaf6f） proc（11333ba70） pid（8913188/8913188） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.650: [ CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.650: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.650: [ CSSD][1029]clssgmRegisterClient: proc（64/11333ba70）, client（1/112e8e550）
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmQueueShare: （11328ed90） target global grock DG_DATA member 1 type 1 queued from client （112e8e550）, global grock DG_DATA, refcount 16
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 16
2012-06-04 23:26:30.766: [ CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.766: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.766: [ CSSD][1029]clssgmRegisterClient: proc（64/11333ba70）, client（2/11328eff0）
2012-06-04 23:26:30.772: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.772: [ CSSD][1029]clssgmQueueShare: （11328f190） target global grock DB+ASM member 1 type 1 queued from client （11328eff0）, global grock DB+ASM, refcount 18
2012-06-04 23:26:30.773: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 18
2012-06-04 23:26:31.382: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:31.382: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:34.394: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 145718 msecs
2012-06-04 23:26:35.392: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:35.392: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:36.396: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 47678 msecs
2012-06-04 23:26:38.571: [ CSSD][2834]clssnmvDiskCheck: （/dev/rhdisk1） No I/O completed after 75% maximum time, 200000 ms, will be considered unusable in 49104 ms
2012-06-04 23:26:39.408: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:39.409: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:42.413: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 153738 msecs
2012-06-04 23:26:43.413: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:43.413: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:44.414: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 55696 msecs
2012-06-04 23:26:47.421: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:47.421: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:50.433: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 161758 msecs
2012-06-04 23:26:51.431: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:51.431: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:52.439: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 63721 msecs
2012-06-04 23:26:55.442: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:55.442: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:58.445: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 169771 msecs
2012-06-04 23:26:59.455: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:59.455: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:00.493: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 71776 msecs
2012-06-04 23:27:03.470: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:03.470: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:06.473: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 177799 msecs
2012-06-04 23:27:07.475: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:07.476: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDeadProc: proc 112f35f50
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDestroyProc: cleaning up proc（112f35f50） con（2ea340） skgpid ospid 10551654 with 0 clients, refcount 0
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea340
2012-06-04 23:27:08.192: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 79475 msecs
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDeadProc: proc 1131b4b10
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDestroyProc: cleaning up proc（1131b4b10） con（2ea3c4） skgpid ospid 10355074 with 0 clients, refcount 0
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea3c4
2012-06-04 23:27:08.569: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.569: [ CSSD][1029]clssgmAllocProc: （1131fe670） allocated
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1131fe670 - 0,1,2,3,4
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb1f8） proc（1131fe670） pid（10551656） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.637: [ CSSD][2834]clssnmvDiskCheck: （/dev/rhdisk1） No I/O completed after 90% maximum time, 200000 ms, will be considered unusable in 19038 ms
2012-06-04 23:27:08.809: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.810: [ CSSD][1029]clssgmAllocProc: （1132593f0） allocated
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132593f0 - 0,1,2,3,4
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb278） proc（1132593f0） pid（10355076） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.855: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.855: [ CSSD][1029]clssgmAllocProc: （113259730） allocated
2012-06-04 23:27:08.857: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113259730 - 0,1,2,3,4
2012-06-04 23:27:08.858: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb2ec） proc（113259730） pid（8192508/8192508） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.858: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.867: [ CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.867: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.867: [ CSSD][1029]clssgmRegisterClient: proc（65/113259730）, client（1/112f35f50）
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmQueueShare: （11334b430） target global grock DG_DATA member 1 type 1 queued from client （112f35f50）, global grock DG_DATA, refcount 17
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 17
2012-06-04 23:27:08.872: [ CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.872: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.872: [ CSSD][1029]clssgmRegisterClient: proc（65/113259730）, client（2/113215ed0）
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmQueueShare: （11334b690） target global grock DB+ASM member 1 type 1 queued from client （113215ed0）, global grock DB+ASM, refcount 19
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 19
2012-06-04 23:27:09.000: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:09.000: [ CSSD][1029]clssgmAllocProc: （11334b8f0） allocated
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11334b8f0 - 0,1,2,3,4
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb37d） proc（11334b8f0） pid（8061184/8061184） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:09.010: [ CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.010: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.010: [ CSSD][1029]clssgmRegisterClient: proc（66/11334b8f0）, client（1/1131b5310）
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmQueueShare: （11334bc30） target global grock DG_DATA member 1 type 1 queued from client （1131b5310）, global grock DG_DATA, refcount 18
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 18
2012-06-04 23:27:09.014: [ CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.014: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.014: [ CSSD][1029]clssgmRegisterClient: proc（66/11334b8f0）, client（2/11334be90）
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmQueueShare: （113350230） target global grock DB+ASM member 1 type 1 queued from client （11334be90）, global grock DB+ASM, refcount 20
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 20
2012-06-04 23:27:11.480: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:11.480: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:14.494: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 185820 msecs
2012-06-04 23:27:15.500: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:15.500: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:16.504: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 87787 msecs
2012-06-04 23:27:19.519: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:19.519: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:20.285: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.285: [ CSSD][1029]clssgmAllocProc: （1132b9290） allocated
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b9290 - 0,1,2,3,4
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb49d） proc（1132b9290） pid（8257826） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:20.299: [ CSSD][1029]clssnmGetNodeNumber: pt1rac2
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDeadProc: proc 1132b9290
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDestroyProc: cleaning up proc（1132b9290） con（2eb49d） skgpid ospid 8257826 with 0 clients, refcount 0
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb49d
2012-06-04 23:27:20.999: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.999: [ CSSD][1029]clssgmAllocProc: （1132b2650） allocated
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 0,1,2,3,4
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb51a） proc（1132b2650） pid（7733742） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:21.005: [ CSSD][1029]clssnmGetNodeNumber: pt1rac1
2012-06-04 23:27:21.289: [ CSSD][1029]clssgmDeadProc: proc 1132b2650
2012-06-04 23:27:21.290: [ CSSD][1029]clssgmDestroyProc: cleaning up proc（1132b2650） con（2eb51a） skgpid ospid 7733742 with 0 clients, refcount 0
2012-06-04 23:27:21.290: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb51a
2012-06-04 23:27:22.001: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 193327 msecs
2012-06-04 23:27:23.533: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:23.534: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDeadProc: proc 113248030
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDestroyProc: cleaning up proc（113248030） con（2ea626） skgpid ospid 9699686 with 0 clients, refcount 0
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea626
2012-06-04 23:27:25.142: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.143: [ CSSD][1029]clssgmAllocProc: （1132b2650） allocated
2012-06-04 23:27:25.143: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 96425 msecs
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 1,2,3,4
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb5c3） proc（1132b2650） pid（9699688） version 11:2:1:4, properties: 1,2,3,4
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.323: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.324: [ CSSD][1029]clssgmAllocProc: （1132b7e50） allocated
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b7e50 - 0,1,2,3,4
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: Connect from con（2eb63a） proc（1132b7e50） pid（7733746/7733746） version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.485: [ CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.485: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.485: [ CSSD][1029]clssgmRegisterClient: proc（67/1132b7e50）, client（1/113153d90）
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmQueueShare: （113338450） target global grock DG_DATA member 1 type 1 queued from client （113153d90）, global grock DG_DATA, refcount 19
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 19
2012-06-04 23:27:25.495: [ CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.495: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.495: [ CSSD][1029]clssgmRegisterClient: proc（67/1132b7e50）, client（2/1132b42b0）
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmQueueShare: （11334c0f0） target global grock DB+ASM member 1 type 1 queued from client （1132b42b0）, global grock DB+ASM, refcount 21
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 21
2012-06-04 23:27:27.542: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:27.542: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:27.680: [ CSSD][2834]（:CSSNM00058:）clssnmvDiskCheck: No I/O completions for 200006 ms for voting file /dev/rhdisk1）
2012-06-04 23:27:27.693: [ CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/rhdisk1 now offline
2012-06-04 23:27:27.693: [ CSSD][2834]（:CSSNM00018:）clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2012-06-04 23:27:27.694: [ CSSD][2834]###################################
2012-06-04 23:27:27.694: [ CSSD][2834]clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.694: [ CSSD][2834]###################################
2012-06-04 23:27:27.694: [ CSSD][2834]（:CSSSC00012:）clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2012-06-04 23:27:27.695: [ CSSD][2834]

----- Call Stack Trace -----
2012-06-04 23:27:27.723: [ CSSD][2834]calling call entry argument values in hex
2012-06-04 23:27:27.723: [ CSSD][2834]location type point （？ means dubious value）
2012-06-04 23:27:27.723: [ CSSD][2834]-------------------- -------- -------------------- ----------------------------
2012-06-04 23:27:27.724: [ CSSD][2834]clssscExit（）+692 bl 100067114 67800000678 ？ 1000EB40C ？
2012-06-04 23:27:27.724: [ CSSD][2834] 110091190 ？ 900000000BA4D14 ？
2012-06-04 23:27:27.724: [ CSSD][2834] 000000000 ？ 013960031 ？
2012-06-04 23:27:27.724: [ CSSD][2834] 013960031 ？ 000000000 ？
2012-06-04 23:27:27.724: [ CSSD][2834]clssnmvDiskCheck（）+ call clssscExit（） 111E38B90 ？ 200000002 ？
2012-06-04 23:27:27.724: [ CSSD][2834]2576 000000001 ？ 1000F52A0 ？
2012-06-04 23:27:27.724: [ CSSD][2834] 110091190 ？ 013960031 ？
2012-06-04 23:27:27.724: [ CSSD][2834] 013960031 ？ 000000000 ？
2012-06-04 23:27:27.724: [ CSSD][2834]clssnmvDiskPingMoni call clssnmvDiskCheck（） 111E38B90 ？ 110BF7FB0 ？
2012-06-04 23:27:27.724: [ CSSD][2834]torThread（）+392 112468680 ？ 000000000 ？
2012-06-04 23:27:27.724: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 013960031 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834]clssscthrdmain（）+20 call clssnmvDiskPingMoni 111E38B90 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834]8 torThread（） 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834]_pthread_body（）+240 call clssscthrdmain（） 111E38B90 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834]fffffffffffffffc call _pthread_body（） 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ？ 000000000 ？
2012-06-04 23:27:27.726: [ CSSD][2834]
2012-06-04 23:27:27.726: [ CSSD][2834]--------------------- Binary Stack Dump ---------------------
2012-06-04 23:27:27.726: [ CSSD][2834]
2012-06-04 23:27:27.726: [ CSSD][2834]========== FRAME [1] （clssscExit（）+692 -> 100067114） ==========
2012-06-04 23:27:27.726: [ CSSD][2834]defined by frame pointers 0x1124683f0 and 0x112468260
2012-06-04 23:27:27.726: [ CSSD][2834]CALL TYPE: bl ERROR SIGNALED: no CALLER: clssscExit
2012-06-04 23:27:27.726: [ CSSD][2834]
Dump of memory from 0x112468260 to 0x1124683f0
2012-06-04 23:27:27.726: [ CSSD][2834]112468260 00000001 124683F0 44000224 000003D7 [.....F..D..$....]
2012-06-04 23:27:27.726: [ CSSD][2834]112468270 00000001 000031D8 00000000 00000083 [......1.........]
2012-06-04 23:27:27.726: [ CSSD][2834]112468280 00000000 00000029 00000001 100182B0 [.......）........]
2012-06-04 23:27:27.726: [ CSSD][2834]112468290 00000678 00000678 00000001 000EB40C [...x...x........]
2012-06-04 23:27:27.726: [ CSSD][2834]1124682A0 00000001 10091190 09000000 00BA4D14 [..............M.]
2012-06-04 23:27:27.726: [ CSSD][2834]1124682B0 00000000 00000000 00000000 13960031 [...............1]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682C0 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682D0 00000001 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682E0 00000000 00000001 43525300 12468368 [........CRS..F.h]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682F0 00000001 00000000 00000001 100181F0 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468300 00000001 10018208 00000001 100181F0 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468310 00000001 10018208 00000001 11E38B90 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468320 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468330 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.728: [ CSSD][2834] Repeat 3 times
2012-06-04 23:27:27.728: [ CSSD][2834]112468370 00000000 00000000 00000000 13960031 [...............1]
2012-06-04 23:27:27.728: [ CSSD][2834]112468380 09000000 00C27160 00000000 00000000 [......q`........]
2012-06-04 23:27:27.728: [ CSSD][2834]112468390 00000001 1002BD70 00000001 117B6590 [.......p.....{e.]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683A0 00000001 00000001 00000000 00000000 [................]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683B0 00000000 00000000 00000646 00000646 [...........F...F]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683C0 00000001 12468428 00000001 117C6C84 [.....F.（.....|l.]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683D0 00000001 117C6C84 00000001 1008FF50 [.....|l........P]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683E0 00000000 00000000 00000001 000F24E8 [..............$.]
2012-06-04 23:27:27.728: [ CSSD][2834]========== FRAME [2] （clssnmvDiskCheck（）+2576 -> clssscExit（）） ==========
2012-06-04 23:27:27.728: [ CSSD][2834]defined by frame pointers 0x112468600 and 0x1124683f0
2012-06-04 23:27:27.729: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssnmvDiskCheck
2012-06-04 23:27:27.729: [ CSSD][2834]
Dump of memory from 0x1124683f0 to 0x112468600
2012-06-04 23:27:27.729: [ CSSD][2834]1124683F0 00000001 12468600 09001000 A0688640 [.....F.......h.@]
2012-06-04 23:27:27.729: [ CSSD][2834]112468400 00000001 0002CD94 4400B22B 00000000 [........D..+....]
2012-06-04 23:27:27.729: [ CSSD][2834]112468410 00000001 001058B8 00000001 100182B0 [......X.........]
2012-06-04 23:27:27.729: [ CSSD][2834]112468420 00000001 11E38B90 00000002 00000002 [................]
2012-06-04 23:27:27.729: [ CSSD][2834]112468430 00000000 00000001 00000001 000F52A0 [..............R.]
2012-06-04 23:27:27.729: [ CSSD][2834]112468440 00000001 10091190 00000000 13960031 [...............1]
2012-06-04 23:27:27.729: [ CSSD][2834]112468450 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.730: [ CSSD][2834]112468460 00000001 1008FF50 00000001 118508F0 [.......P........]
2012-06-04 23:27:27.730: [ CSSD][2834]112468470 00000001 117B6590 8C91440C 00030D46 [.....{e...D....F]
2012-06-04 23:27:27.730: [ CSSD][2834]112468480 00000001 12C508B0 00000000 00030D40 [...............@]
2012-06-04 23:27:27.730: [ CSSD][2834]112468490 00010001 9DD81012 D0E54F97 BFEE0F2C [..........O....,]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684A0 AF8AA88C 00000000 2200088B 00000000 [........".......]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684B0 09000000 014F0814 09001000 A0688640 [.....O.......h.@]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684C0 00030D40 1045F530 00000004 00000004 [...@.E.0........]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684D0 00000001 00105DE8 00000001 000F2BD4 [......].......+.]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684E0 00000001 000F2BD4 00030D40 000F24E8 [......+....@..$.]
2012-06-04 23:27:27.731: [ CSSD][2834]1124684F0 00000001 00000001 2200022B 00000000 [........"..+....]
2012-06-04 23:27:27.731: [ CSSD][2834]112468500 00000000 00000000 FFFFFFF5 000F24E8 [..............$.]
2012-06-04 23:27:27.731: [ CSSD][2834]112468510 00000001 12468600 09001000 A0688640 [.....F.......h.@]
2012-06-04 23:27:27.731: [ CSSD][2834]112468520 00000001 00070DFC 00000001 117C6C84 [.............|l.]
2012-06-04 23:27:27.731: [ CSSD][2834]112468530 00000001 117C6C84 00000001 100182B0 [.....|l.........]
2012-06-04 23:27:27.731: [ CSSD][2834]112468540 00000001 1045F530 00000004 00000004 [.....E.0........]
2012-06-04 23:27:27.731: [ CSSD][2834]112468550 00000001 00105DE8 00000000 00000009 [......].........]
2012-06-04 23:27:27.731: [ CSSD][2834]112468560 00000000 000003E8 00000000 000003E8 [................]
2012-06-04 23:27:27.731: [ CSSD][2834]112468570 00000000 13960031 00000000 00000040 [.......1.......@]
2012-06-04 23:27:27.732: [ CSSD][2834]112468580 00000001 1008FF50 00000001 117B6590 [.......P.....{e.]
2012-06-04 23:27:27.732: [ CSSD][2834]112468590 00000001 1002BD70 00000001 117B65A0 [.......p.....{e.]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685A0 00000001 117B66A0 FFFFFFB2 000003D4 [.....{f.........]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685B0 00000001 117C6C84 00000001 117C6C84 [.....|l......|l.]
2012-06-04 23:27:27.732: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.732: [ CSSD][2834]1124685D0 00000001 000F2BD4 00000001 000F2BD4 [......+.......+.]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685E0 00000000 00000000 00000001 12C508B0 [................]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685F0 00000000 00000000 00000001 000F24E8 [..............$.]
2012-06-04 23:27:27.732: [ CSSD][2834]========== FRAME [3] （clssnmvDiskPingMonitorThread（）+392 -> clssnmvDiskCheck（）） ==========
2012-06-04 23:27:27.732: [ CSSD][2834]defined by frame pointers 0x1124686f0 and 0x112468600
2012-06-04 23:27:27.732: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.732: [ CSSD][2834]
Dump of memory from 0x112468600 to 0x1124686f0
2012-06-04 23:27:27.733: [ CSSD][2834]112468600 00000001 124686F0 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468610 00000001 0002CFAC 09001000 A0688640 [.............h.@]
2012-06-04 23:27:27.733: [ CSSD][2834]112468620 00000000 00000000 00000001 100182B0 [................]
2012-06-04 23:27:27.733: [ CSSD][2834]112468630 00000001 11E38B90 00000001 10BF7FB0 [................]
2012-06-04 23:27:27.733: [ CSSD][2834]112468640 00000001 12468680 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468650 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.733: [ CSSD][2834]112468660 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590 [.............{e.]
2012-06-04 23:27:27.733: [ CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0 [...@..@$........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468690 09000000 014F0814 00000000 00000000 [.....O..........]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686A0 00000000 00000000 00000000 00000001 [................]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8 [................]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78 [.............F.x]
2012-06-04 23:27:27.734: [ CSSD][2834]========== FRAME [4] （clssscthrdmain（）+208 -> clssnmvDiskPingMonitorThread（）） ==========
2012-06-04 23:27:27.734: [ CSSD][2834]defined by frame pointers 0x112468790 and 0x1124686f0
2012-06-04 23:27:27.734: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssscthrdmain
2012-06-04 23:27:27.734: [ CSSD][2834]
Dump of memory from 0x1124686f0 to 0x112468790
2012-06-04 23:27:27.734: [ CSSD][2834]1124686F0 00000001 12468790 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.734: [ CSSD][2834]112468700 00000001 0000A294 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468710 00000000 00000000 00000001 100182B0 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468720 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468730 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.735: [ CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.735: [ CSSD][2834]112468770 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468780 00000000 00000000 00000001 1008FF50 [...............P]
2012-06-04 23:27:27.735: [ CSSD][2834]========== FRAME [5] （_pthread_body（）+240 -> clssscthrdmain（）） ==========
2012-06-04 23:27:27.735: [ CSSD][2834]defined by frame pointers 0x112468810 and 0x112468790
2012-06-04 23:27:27.735: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: _pthread_body
2012-06-04 23:27:27.735: [ CSSD][2834]
Dump of memory from 0x112468790 to 0x112468890
2012-06-04 23:27:27.736: [ CSSD][2834]112468790 00000001 12468810 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687A0 09000000 00B7BD54 00000000 00000000 [.......T........]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687B0 00000000 00000000 09001000 A0288350 [.............（.P]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687C0 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687D0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.737: [ CSSD][2834] Repeat 11 times
2012-06-04 23:27:27.737: [ CSSD][2834]========== FRAME [6] （fffffffffffffffc -> _pthread_body（）） ==========
2012-06-04 23:27:27.737: [ CSSD][2834]defined by frame pointers 0x0 and 0x112468810
2012-06-04 23:27:27.737: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: fffffffffffffffc
2012-06-04 23:27:27.737: [ CSSD][2834]
----- Argument/Register Address Dump -----

2012-06-04 23:27:27.737: [ CSSD][2834]Argument/Register addr=0x1000eb40c.
Dump of memory from 0x1000eb3cc to 0x1000eb50c
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3C0 6C206572 [l er]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3D0 726F7220 6F636375 72726564 20616E64 [ror occurred and]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3E0 20746865 20435353 20646165 6D6F6E20 [ the CSS daemon ]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3F0 69732074 65726D69 6E617469 6E672061 [is terminating a]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB400 626E6F72 6D616C6C 79004942 283A4353 [bnormally.IB（:CS]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB410 53534330 30303132 3A290049 636C7373 [SSC00012:）.Iclss]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB420 73634578 69743A20 53746172 74696E67 [scExit: Starting]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB430 20435253 4420636C 65616E75 70004942 [ CRSD cleanup.IB]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB440 636C7373 73634578 69743A20 43525344 [clssscExit: CRSD]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB450 20636C65 616E7570 20737461 74757320 [ cleanup status ]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB460 25640049 636C7373 73634578 69743A20 [%d.IclssscExit: ]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB470 43525344 20636C65 616E7570 20666169 [CRSD cleanup fai]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB480 6C656420 77697468 20256400 636C7373 [led with %d.clss]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB490 73634578 69743A20 43525344 20636C65 [scExit: CRSD cle]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB4A0 616E7570 20737563 63657373 66756C6C [anup successfull]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4B0 7920636F 6D706C65 74656400 636C7373 [y completed.clss]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4C0 73634578 69743A20 61626F72 7420616C [scExit: abort al]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4D0 72656164 79207365 74202564 0049424D [ready set %d.IBM]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4E0 41535345 52542025 73202564 0049424D [ASSERT %s %d.IBM]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4F0 636C7373 73632E63 0049424D 283A4353 [clsssc.c.IBM（:CS]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB500 53534330 30303133 3A29636C [SSC00013:）cl]
2012-06-04 23:27:27.739: [ CSSD][2834]Argument/Register addr=0x110091190.
Dump of memory from 0x110091150 to 0x110091290
2012-06-04 23:27:27.739: [ CSSD][2834]110091150 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.739: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.739: [ CSSD][2834]110091180 00000000 00000000 00000001 00000001 [................]
2012-06-04 23:27:27.740: [ CSSD][2834]110091190 2F736F66 742F6772 69642F70 726F6475 [/soft/grid/produ]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911A0 63742F31 312E322E 302E322F 6769682F [ct/11.2.0.2/gih/]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911B0 6C6F672F 70743172 6163322F 63737364 [log/pt1rac2/cssd]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911C0 2F6F6373 73642E6C 6F670000 00000000 [/ocssd.log......]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911D0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ CSSD][2834] Repeat 11 times
2012-06-04 23:27:27.741: [ CSSD][2834]Argument/Register addr=0x900000000ba4d14.
Dump of memory from 0x900000000ba4cd4 to 0x900000000ba4e14
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CD0 38600000 4BFD5608 00000000 [8`..K.V.....]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CE0 000C0000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CF0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D00 348D8278 41810008 38800000 E8630000 [4..xA...8....c..]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D10 38A00008 81840230 7C0004AC 398CFFFF [8......0|...9...]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D20 E9040110 38E00000 65081000 7D251828 [....8...e...}%.（]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D30 7C894000 40860014 7CE5192D 40A2FFF0 [|.@.@...|..-@...]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D40 91840230 4E800020 7C632A14 4BFD4694 [...0N.. |c*.K.F.]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D50 00000000 000C0000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D60 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D80 7D251828 64871000 2C890000 40860014 [}%.（d...,...@...]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D90 7CE5192D 4C00012C 4D820020 4BFFFFE4 [|..-L..,M.. K...]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DA0 7C862378 7C851A14 7CC53378 4BFD5534 [|.#x|...|.3xK.U4]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DB0 00000000 000C0000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DC0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.743: [ CSSD][2834] Repeat 3 times
2012-06-04 23:27:27.743: [ CSSD][2834]900000000BA4E00 7D251828 64871000 2C890000 40860014 [}%.（d...,...@...]
2012-06-04 23:27:27.743: [ CSSD][2834]900000000BA4E10 7CE5192D [|..-]
2012-06-04 23:27:27.743: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.743: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.744: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.744: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.744: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.745: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.745: [ CSSD][2834]Argument/Register addr=0x111e38b90.
Dump of memory from 0x111e38b50 to 0x111e38c90
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B50 00000001 00000001 8C8FC179 000003E8 [...........y....]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B60 8C914038 8C913A22 0000000B 00000000 [..@8..:"........]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B70 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B80 5B5B5B5B 00000000 00000000 000001E0 [[[[[............]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B90 00000001 1008FF50 00000001 10BF7FB0 [.......P........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BA0 00000001 11E3A320 00000001 11E389A0 [....... ........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BB0 00000B12 00000B12 636C7373 6E6D7644 [........clssnmvD]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BC0 69736B50 696E674D 6F6E6974 6F725468 [iskPingMonitorTh]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BD0 72656164 00000000 00000000 00000000 [read............]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BE0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BF0 00000000 00000000 00020000 00000000 [................]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C00 00000001 1045F530 00000001 11E38B90 [.....E.0........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C10 00000001 100179B0 00000000 00000000 [......y.........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C20 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.747: [ CSSD][2834] Repeat 4 times
2012-06-04 23:27:27.747: [ CSSD][2834]111E38C70 00000000 00000001 00000000 00000005 [................]
2012-06-04 23:27:27.747: [ CSSD][2834]111E38C80 00000000 00001677 00000000 00000000 [.......w........]
2012-06-04 23:27:27.747: [ CSSD][2834]Argument/Register addr=0x1.
Dump of memory from 0x0 to 0x101
2012-06-04 23:27:27.747: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.748: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.748: [ CSSD][2834]000000100 7C7243A6 [|rC.]
2012-06-04 23:27:27.748: [ CSSD][2834]Argument/Register addr=0x1000f52a0.
Dump of memory from 0x1000f5260 to 0x1000f53a0
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5260 6B436865 636B3A20 41626F72 74696E67 [kCheck: Aborting]
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5270 2C202564 206F6620 25642025 7320766F [, %d of %d %s vo]
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5280 74696E67 20646973 6B732061 7661696C [ting disks avail]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5290 61626C65 2C206E65 65642025 75004942 [able, need %u.IB]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52A0 283A4353 534E4D30 30303138 3A290049 [（:CSSNM00018:）.I]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52B0 636C7373 6E6D7644 69736B50 696E674D [clssnmvDiskPingM]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52C0 6F6E6974 6F725468 72656164 3A20206E [onitorThread: n]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52D0 6F20766F 74696E67 20646973 6B2C0049 [o voting disk,.I]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52E0 636C7373 6E6D7644 69736B50 4D543A20 [clssnmvDiskPMT: ]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52F0 736C6565 70696E67 20666F72 20256420 [sleeping for %d ]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5300 6D730049 636C7373 6E6D7653 63616E43 [ms.IclssnmvScanC]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5310 6F6D706C 6574696F 6E733A20 4661696C [ompletions: Fail]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5320 75726520 2564206E 6F746966 79696E67 [ure %d notifying]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5330 20636C69 656E7473 206F6620 776F726B [ clients of work]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5340 20636F6D 706C6574 696F6E2E 0049424D [ completion..IBM]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5350 636C7373 6E6D7653 63616E43 6F6D706C [clssnmvScanCompl]
2012-06-04 23:27: