Using DBX and KDB to build stack traces

Question

I have a hung process, how can I get a stack trace of it?

Cause



Answer

NOTE: Not all processes that show up in ps -ef will be able to have stack traces built on them. Old processes tend to be eventually paged out of memory and neither dbx or kdb will then be able to be used to look at the stack trace for that process.

DBX Stack Trace Instructions for building a stack trace on a hung process:

In order to use dbx, the customer must first have the fileset bos.adt.debug installed.

Attach to hung process
1. Capture console output, enter:
script

2. Enter:
ps -ef | grep

3. Enter:
dbx -a

4. Format trace, enter:
where

5. Leave dbx, enter:
detach (Typing quit will kill the process)

6. To leave script, type exit.
The script will be named typescript and will be located in the current
working directory.

Steps to obtain thread stack trace using kdb
Using the alog process as an example.

1) Start script session to capture data:
# script /tmp/kdb.out

2) Find the process id and convert it into hexadecimal:

# ps -ef | grep alog
UID PID PPID C STIME TTY TIME CMD
root 1231 1 1 Jun 30 - 1:12 alog

Convert 1231 to Hexadecimal number
1231 converts to 4CF

3) Start kdb

# kdb

4) Locate the process while in kdb

(0) p * | grep 4CF

Ex.
pvproc+013800 78 alog ACTIVE 004E036 004A01E
0000000002525400 0 0001

5) find initial thread
(0) p (PSLOT) | grep pvthread [The pslot is the second
column. In the above example, it is 78]

6) locate initial thread in 'p' output
example:

...
THREAD..... threadlist :EA005E00
...

7) list function stack for initial thread
(0) f pvthread+005E00

8) Exit out of the script session
# exit
Data will be saved in /tmp/kdb.out.

The procstack command can also be used to print the stack of a process.

# ps -ef | grep alog
root 491752 450572 0 15:45:52 pts/4 0:00 alog

# procstack 491752
491752: alog
0xd0375da4 read(??, ??, ??) + 0x1a8
0x10001500 main(??, ??) + 0x11b0
0x10000198 __start() + 0x98