Which Process Is Using Up Most The CPU Resources

Question

How can you determine which process is using up the most CPU time?

Cause



Answer

The following commands and tools can be used to find which process is using the most cpu resources.

1. topas -P



In the topas -P output above the process called "cpu-eater" is the top consumer of cpu resources.

2. tprof -x sleep 10; vi sleep.prof

Using DBX and KDB to build stack traces

Question

I have a hung process, how can I get a stack trace of it?

Cause



Answer

NOTE: Not all processes that show up in ps -ef will be able to have stack traces built on them. Old processes tend to be eventually paged out of memory and neither dbx or kdb will then be able to be used to look at the stack trace for that process.

DBX Stack Trace Instructions for building a stack trace on a hung process:

In order to use dbx, the customer must first have the fileset bos.adt.debug installed.

Attach to hung process
1. Capture console output, enter:
script

2. Enter:
ps -ef | grep

3. Enter:
dbx -a

4. Format trace, enter:
where

5. Leave dbx, enter:
detach (Typing quit will kill the process)

6. To leave script, type exit.
The script will be named typescript and will be located in the current
working directory.

Steps to obtain thread stack trace using kdb
Using the alog process as an example.

1) Start script session to capture data:
# script /tmp/kdb.out

2) Find the process id and convert it into hexadecimal:

# ps -ef | grep alog
UID PID PPID C STIME TTY TIME CMD
root 1231 1 1 Jun 30 - 1:12 alog

Convert 1231 to Hexadecimal number
1231 converts to 4CF

3) Start kdb

# kdb

4) Locate the process while in kdb

(0) p * | grep 4CF

Ex.
pvproc+013800 78 alog ACTIVE 004E036 004A01E
0000000002525400 0 0001

5) find initial thread
(0) p (PSLOT) | grep pvthread [The pslot is the second
column. In the above example, it is 78]

6) locate initial thread in 'p' output
example:

...
THREAD..... threadlist :EA005E00
...

7) list function stack for initial thread
(0) f pvthread+005E00

8) Exit out of the script session
# exit
Data will be saved in /tmp/kdb.out.

The procstack command can also be used to print the stack of a process.

# ps -ef | grep alog
root 491752 450572 0 15:45:52 pts/4 0:00 alog

# procstack 491752
491752: alog
0xd0375da4 read(??, ??, ??) + 0x1a8
0x10001500 main(??, ??) + 0x11b0
0x10000198 __start() + 0x98

Determine which process is using a specific network port on AIX with or without lsof

Method 1. Using lsof.

It will be easy if you have lsof installed.

# lsof -i:32876
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
oracle 135744 oracle7 13u IPv4 0x70bbe200 0t11 UDP loopback:32876

# ps -ef|grep 135744
oracle7 135744 1 0 Apr 21 - 2:45 ora_pmon_ftc_p01

Method 2. Using netstat and rmsock

# netstat -Aan|grep 9991
f100020000626b98 tcp4 0 0 *.9991 *.* LISTEN

# rmsock f100020000626b98 tcpcb
The socket 0x626808 is being held by proccess 200928 (sysscand).

#ps -ef|grep 200928
root 200928 1 0 Apr 21 - 0:01 /opt/sysscan/bin/sysscand


Method 3: Using netstat and kdb.

# netstat -Aan|grep 9991
f100020000626b98 tcp4 0 0 *.9991 *.* LISTEN

# kdb
The specified kernel file is a 64-bit kernel
Preserving 1418431 bytes of symbol table
First symbol __mulh
START END
0000000000001000 0000000003E0D050 start+000FD8
F00000002FF47600 F00000002FFDC940 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F100070F00000000 F100070F10000000 pvproc+000000
F100070F10000000 F100070F18000000 pvthread+000000
PFT:
PVT:
id....................0002
raddr.....0000000000724000 eaddr.....F200800030000000
size..............00040000 align.............00001000
valid..1 ros....0 fixlmb.1 seg....0 wimg...2
ERROR: Unable to acess nfs_syms
(0)> sockinfo f100020000626b98 tcpcb
.................
..............

on last a few lines.

proc/fd: 49/0
proc/fd: fd: 0
SLOT NAME STATE PID PPID ADSPACE CL #THS

pvproc+00C400 49*sysscand ACTIVE 00310E0 0000001 00000000285C7400 0 0001



(0)> hcal 00310E0
Value hexa: 000310E0 Value decimal: 200928

(0)> quit

# ps -ef|grep 200928
root 200928 1 0 Apr 21 - 0:01 /opt/sysscan/bin/sysscand

Script to list process in current runq in AIX

The script also exclude the process created by the script itself.

#!/bin/ksh
myPID=$$
echo
echo "CMD PID Uptime CPU_time THCNT in_RunQ" | awk ' { printf ("%-10s %-10s %-15s %-10s %-10s %s",$1,$2,$3,$4,$5,$6"\n") }'
echo "--------------------------------------------------------------------"
ps -emo pid,tid,comm,etime,time,thcount,ppid,state | awk '$1 != "-" && $7 != MYPID { if (NB_THREAD != 0 ) { print PID"\t"NB_THREAD } ; PID=$3"\t"$1"\t"$4"\t"$5"\t"$6 ; NB_THREAD=0 } ; $NF=="R" { NB_THREAD=NB_THREAD + 1 }' MYPID="$myPID" | awk ' { printf ("%-10s %-10s %-15s %-10s %-10s %s",$1,$2,$3,$4,$5,$6"\n") }'
echo