Craig,
From what I can see, the errors reported in the diag_log table span a very
wide period of time. Most of these errors, including the most recent one,
are from port fffffe50. What I'd do now if it were my dump : format the
config and check which LDEV, if any, owns port fffffe50, and try to
correlate that to an IO_IN_PROGRESS in the io_request_table. I'd also format
the timer_globals and retrieve the TMR_TICKS_PER_MILLISEC field, to check
what's the timing interval between the idag_logs that are reported,
especially toward the top of the pile. From that, I'd give a try at
formatting the port_object for port fffffe50 ... SCSI is a likely suspect,
yes, but there could be others. I've also notices quite a number of
IOS_IN_PROGRESS, which is a little surprise for me, so 'd also check for
possible deadlocks.
My, looks like you may be looking for a needle in a haystack, which is the
biggest fun in dumpreading !!!
Have fun,
Christian
> -----Message d'origine-----
> De : HP-3000 Systems Discussion [mailto:[log in to unmask]]De la
> part de Craig Lalley
> Envoye : mardi 20 janvier 2004 12:43
> A : [log in to unmask]
> Objet : Re: [HP3000-L] Memory Dump
>
>
> Christian Lheureux <[log in to unmask]> wrote:
>
> Hi Craig !
>
> In case nobody replied before me .... you need to issue at least
> io_ios_diag_log, and possibly other commands/macros. Then
> you'll need to
> format the config to retrieve the hardware path.
>
> *******************
>
> Christian,
>
>
> Thank you for your help, I got a bit further. Here is some
> of the output. Can we at least get the offending SCSI
> path/card? Or maybe an LDEV and I will tie it back.
>
>
> To let everyone else know, I am wondering if the SCSI
> firmware is causing a failure in BIGMEM (> 3.75) MB systems.
> The firmware is 3944, latest firmware.
>
> Thanks again Christian!
>
> /* Start of data *************************************
>
> Diag log table resides at: $8064e000
>
> DIAGNOSTIC LOGGING HEADER
> -------------------------
>
> CURRENT INDEX NEXT INDEX MAX INDEX AUTO EXPAND AUTO
> EXPAND BOUND
> ------------- ---------- --------- -----------
> -----------------
> $d $e $54 FALSE $54
>
>
> DIAGNOSTIC ENTRY
> ----------------
>
> ENTRY TIME STAMP FROM PORT PROC ERROR SUBSYSTEM
> ----- ----------------- --------- ----- ----- -------------
> #13 $2aef13aab9ca7a0 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #12 $2aef0a2d70313a2 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #11 $2aef08d8cb267c1 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #10 $2aef088b9638013 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #9 $2aef00b41312330 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #8 $2aeeff522f31a61 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #7 $2aeefef72777640 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #6 $2aeef74a5f3b520 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #5 $2aeef62e06b5c40 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #4 $2aeeee02fad6d80 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #3 $2aeee8da719be51 $ffffff74 #64 #-41 #243 No
> entry in message cat
> alog for warning #32765, subsys #243.
> No entry in message catalog for error #41, subsys #243.
> #2 $2aeee6ace440d30 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #1 $2aeee46999de4e1 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #0 $2aeee08ab968e81 $ffffff74 #64 #-41 #243 No
> entry in message cat
> alog for warning #32765, subsys #243.
> No entry in message catalog for error #41, subsys #243.
>
> $14a ($252) nmdat >
> Diag log table resides at: $8064e000
>
> DIAGNOSTIC LOGGING HEADER
> -------------------------
>
> CURRENT INDEX NEXT INDEX MAX INDEX AUTO EXPAND AUTO
> EXPAND BOUND
> ------------- ---------- --------- -----------
> -----------------
> $d $e $54 FALSE $54
>
>
> DIAGNOSTIC ENTRY
> ----------------
>
> ENTRY TIME STAMP FROM PORT PROC ERROR SUBSYSTEM
> ----- ----------------- --------- ----- ----- -------------
> #13 $2aef13aab9ca7a0 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #12 $2aef0a2d70313a2 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #11 $2aef08d8cb267c1 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #10 $2aef088b9638013 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #9 $2aef00b41312330 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #8 $2aeeff522f31a61 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #7 $2aeefef72777640 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #6 $2aeef74a5f3b520 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #5 $2aeef62e06b5c40 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #4 $2aeeee02fad6d80 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #3 $2aeee8da719be51 $ffffff74 #64 #-41 #243 No
> entry in message cat
> alog for warning #32765, subsys #243.
> No entry in message catalog for error #41, subsys #243.
> #2 $2aeee6ace440d30 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #1 $2aeee46999de4e1 $fffffe50 #13 #0 #109 No
> entry in message cat
> alog for warning #32765, subsys #109.
> No entry in message catalog for warning #0, subsys #109.
> #0 $2aeee08ab968e81 $ffffff74 #64 #-41 #243 No
> entry in message cat
> alog for warning #32765, subsys #243.
> No entry in message catalog for error #41, subsys #243.
>
> $14a ($252) nmdat >
>
> $153 ($252) nmdat > io_request_table
>
> Formatted I/O Request Table Entries
>
> R C
> E M
> A I
> D O
> I
> N R
> RID # G Q STATE FUNC LDEV PIN STATUS PRI
> ----- - - ---------------- ---- ---- ----- --------- -----
>
> $1 IOS_COMPLETED $4 12 $252 $0 $1bff
> $2 IOS_COMPLETED $4 9 $267 $0 $1bff
> $3 IOS_INPROGRESS $8 6 $49 $0 $4dff
> $4 T IOS_QUEUED $1 17 $245 $0 $2e7f
> $5 IOS_COMPLETED $4 9 $267 $0 $1bff
> $6 IOS_COMPLETED $4 9 $267 $0 $1bff
> $7 IOS_INPROGRESS $4 9 $267 $0 $1bff
> $8 IOS_COMPLETED $4 12 $252 $0 $1bff
> $9 IOS_COMPLETED $4 11 $33b $0 $1bff
> $a T IOS_QUEUED $1 192 $2fa $0 $1dbf
> $b IOS_COMPLETED $4 11 $33b $0 $1bff
> $c IOS_QUEUED $4 11 $33b $0 $1bff
> $d IOS_COMPLETED $4 9 $267 $0 $1bff
> $e IOS_COMPLETED $4 12 $252 $0 $1bff
> $f IOS_COMPLETED $4 12 $252 $0 $1bff
> $10 IOS_COMPLETED $4 9 $267 $0 $1bff
> $11 IOS_COMPLETED $4 9 $267 $0 $1bff
> $12 T IOS_QUEUED $1 14 $263 $0 $33ff
> $13 T IOS_QUEUED $1 20 $4b $0 $33ff
> $14 IOS_QUEUED $4 9 $267 $0 $1bff
> $15 IOS_INPROGRESS $4 11 $33b $0 $1bff
> $16 IOS_COMPLETED $4 12 $252 $0 $1bff
> $17 IOS_COMPLETED $4 12 $252 $0 $1bff
> $18 IOS_COMPLETED $4 11 $33b $0 $1bff
> $19 IOS_COMPLETED $4 12 $252 $0 $1bff
> $1a IOS_COMPLETED $4 9 $267 $0 $1bff
> $1b IOS_COMPLETED $4 12 $252 $0 $1bff
> $1c IOS_COMPLETED $4 11 $33b $0 $1bff
> $1d IOS_COMPLETED $4 11 $33b $0 $1bff
> $1e IOS_COMPLETED $4 11 $33b $0 $1bff
> $1f IOS_COMPLETED $4 9 $267 $0 $1bff
> $20 T IOS_QUEUED $1 123 $347 $0 $33bf
> $21 IOS_COMPLETED $4 12 $252 $0 $1bff
> $22 T IOS_QUEUED $1 28 $34b $0 $1bff
> $23 IOS_COMPLETED $4 12 $252 $0 $1bff
> $24 IOS_COMPLETED $4 11 $33b $0 $1bff
> $25 IOS_COMPLETED $4 12 $252 $0 $1bff
> $26 IOS_COMPLETED $4 12 $252 $0 $1bff
> $27 IOS_COMPLETED $4 12 $252 $0 $1bff
> $28 IOS_COMPLETED $4 11 $33b $0 $1bff
> $29 IOS_COMPLETED $4 9 $267 $0 $1bff
> $2a IOS_COMPLETED $4 12 $252 $0 $1bff
> $2b IOS_COMPLETED $4 11 $33b $0 $1bff
> $2c IOS_COMPLETED $4 9 $267 $0 $1bff
> $2d IOS_COMPLETED $4 9 $267 $0 $1bff
> $2e IOS_COMPLETED $4 9 $267 $0 $1bff
> $2f IOS_COMPLETED $4 12 $252 $0 $1bff
> $30 IOS_INPROGRESS $4 11 $33b $0 $1bff
> $31 IOS_COMPLETED $4 12 $252 $0 $1bff
> $32 IOS_COMPLETED $4 12 $252 $0 $1bff
> $33 IOS_COMPLETED $4 12 $252 $0 $1bff
> $34 IOS_COMPLETED $4 11 $33b $0 $1bff
> $35 IOS_COMPLETED $4 12 $252 $0 $1bff
> $36 IOS_COMPLETED $4 12 $252 $0 $1bff
> $37 IOS_COMPLETED $4 11 $33b $0 $1bff
> $38 T IOS_QUEUED $1 59 $3a6 $0 $333f
> $39 IOS_COMPLETED $4 12 $252 $0 $1bff
> $3a IOS_COMPLETED $4 12 $252 $0 $1bff
> $3b T IOS_QUEUED $1 77 $2de $0 $327f
> $3c IOS_COMPLETED $4 12 $252 $0 $1bff
> $3d IOS_COMPLETED $4 12 $252 $0 $1bff
> $3e IOS_COMPLETED $4 12 $252 $0 $1bff
> $3f IOS_COMPLETED $4 11 $33b $0 $1bff
> $40 T IOS_QUEUED $1 227 $32f $0 $23ff
> $41 IOS_COMPLETED $4 12 $252 $0 $1bff
> $42 IOS_COMPLETED $4 11 $33b $0 $1bff
> $43 IOS_COMPLETED $4 12 $252 $0 $1bff
> $44 IOS_COMPLETED $4 12 $252 $0 $1bff
> $45 IOS_COMPLETED $4 12 $252 $0 $1bff
> $46 IOS_COMPLETED $4 12 $252 $0 $1bff
> $47 IOS_COMPLETED $4 11 $33b $0 $1bff
> $48 IOS_COMPLETED $4 11 $33b $0 $1bff
> $49 IOS_COMPLETED $4 11 $33b $0 $1bff
> $4a IOS_COMPLETED $4 9 $267 $0 $1bff
> $4b T IOS_QUEUED $1 5 $317 $0 $1bff
> $4c IOS_COMPLETED $4 12 $252 $0 $1bff
> $4d IOS_COMPLETED $4 9 $267 $0 $1bff
> $4e IOS_COMPLETED $4 9 $267 $0 $1bff
> $4f IOS_COMPLETED $4 11 $33b $0 $1bff
> $50 IOS_INPROGRESS $4 9 $267 $0 $1bff
> $51 IOS_COMPLETED $4 11 $33b $0 $1bff
> $52 T IOS_QUEUED $1 75 $219 $0 $337f
> $53 T IOS_QUEUED $1 121 $310 $0 $1bff
> $54 T IOS_QUEUED $1 74 $258 $0 $1bff
> $55 IOS_COMPLETED $4 12 $252 $0 $1bff
> $56 IOS_COMPLETED $4 12 $252 $0 $1bff
> $57 IOS_COMPLETED $4 12 $252 $0 $1bff
> $58 IOS_COMPLETED $4 12 $252 $0 $1bff
> $59 IOS_COMPLETED $4 12 $252 $0 $1bff
> $5a IOS_COMPLETED $4 12 $252 $0 $1bff
> $5b IOS_COMPLETED $4 12 $252 $0 $1bff
> $5c IOS_COMPLETED $4 12 $252 $0 $1bff
> $5d IOS_COMPLETED $4 11 $33b $0 $1bff
> $5e IOS_COMPLETED $4 11 $33b $0 $1bff
> $5f T IOS_QUEUED $1 246 $363 $0 $33ff
> $60 IOS_COMPLETED $4 11 $33b $0 $1bff
> $61 IOS_QUEUED $4 9 $267 $0 $1bff
> $62 IOS_COMPLETED $4 9 $267 $0 $1bff
> $63 IOS_COMPLETED $4 12 $252 $0 $1bff
> $64 T IOS_QUEUED $1 29 $26c $0 $33ff
> $65 IOS_QUEUED $4 11 $33b $0 $1bff
> $66 IOS_COMPLETED $4 9 $267 $0 $1bff
> $67 IOS_COMPLETED $4 12 $252 $0 $1bff
> $68 IOS_COMPLETED $4 11 $33b $0 $1bff
> $69 IOS_COMPLETED $4 12 $252 $0 $1bff
> $6a T IOS_QUEUED $1 125 $2c3 $0 $1bff
> $6b IOS_COMPLETED $4 11 $33b $0 $1bff
> $6c T IOS_QUEUED $1 16 $3a7 $0 $33ff
> $6d IOS_COMPLETED $4 9 $267 $0 $1bff
> $6e IOS_COMPLETED $4 11 $33b $0 $1bff
>
> $154 ($252) nmdat >
>
> **************** *************************
> OS STATUS VALUES IO SERVICES STATUS VALUES
> **************** *************************
>
> Info Subsys Error # Proc # Port # Subsys
> ---- ------ ------- ------ ------ ------
> $0 $0 $50 $ffa1 $ffffff7f
> IO Services
> $0 $0 $61 $ffbd $0
> IO Services
> $0 $0 $ffcf $fff7 $0
> IO Services
> $0 $0 $ffcf $fff7 $0
> IO Services
> $0 $0 $ffcf $fff7 $0
> IO Services
> $0 $0 $26 $fff7 $0
> IO Services
> $0 $0 $26 $fff7 $0
> IO Services
> $0 $0 $26 $fff7 $0
> IO Services
> $0 $0 $26 $fff7 $0
> IO Services
> $0 $0 $76 $ffd3 $0
> IO Services
> $0 $0 $76 $ffd3 $0
> IO Services
> $0 $0 $ffda $fff7 $0
> IO Services
> $0 $0 $ffda $fff7 $0
> IO Services
> $0 $0 $ffda $fff7 $0
> IO Services
> $0 $0 $9 $ffcc $0
> IO Services
>
>
>
>
> ---------------------------------
> Do you Yahoo!?
> Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
>
> * To join/leave the list, search archives, change list settings, *
> * etc., please visit http://raven.utc.edu/archives/hp3000-l.html *
* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *
|
