Your Ad Here

IBM AIX/UNIX system storage administration ksh/perl scripting

Tuesday, March 16, 2010

Command Respawning Too Rapidly From init, wtmp Filling Up




 Technote (FAQ)
Question
How can I find out what command is respawning too rapidly and filling up my wtmp file?

Answer
A common situation is that the init process is repeatedly attempting to start a failing process. The init man page describes what happens when init finds an entry is being respawned:

If the init command finds that it is continuously running an entry in

the /etc/inittab file (more than five times in 225 seconds), it assumes

that an error in the entry command string exists. It then prints an

error message to the console and logs an error in the system error log.

After the message is sent, the entry does not run for 60 seconds. If

the error continues to occur, the command will respawn the entry only

five times every 240 seconds. The init command continues to assume an

error occurred until the command does not respond five times in the

interval, or until it receives a signal from a user. The init command

logs an error for only the first occurrence of the error.
To find out what is being respawned use the steps below.

1. Check the console or console logs

Check on the console to see if init is writing an error message similar to the one below:

0 Thu Jan 22 10:16:27 EST 2009

INIT: Command is respawning too rapidly. Check for possible errors.

id:  xvfb "/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &"


Or search through the console log using the alog command:
# alog -t console -o | more
2. Check errpt
Next there may be an entry in the errpt output with the label "INIT_RAPID" like below:

LABEL: INIT_RAPID

IDENTIFIER: 3A30359F


Date/Time:       Wed Jan 28 10:14:17 2009

Sequence Number: 1789

Machine Id:      00CC2F914C00

Node Id:         libgng

Class:           S

Type:            TEMP

Resource Name:   init


Description

SOFTWARE PROGRAM ERROR


Probable Causes

SOFTWARE PROGRAM


User Causes

PERFORMANCE DEGRADED


Recommended Actions

REVIEW DETAILED DATA


Detail Data

SOFTWARE ERROR CODE

Command is respawning too rapidly. Check for possible errors.

COMMAND

id:  xvfb "/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &"

Both messages clearly identify the failing command that is being run out of the /etc/inittab file.


3. Check the wtmp file
If the warning messages are not noticed on the system console or in errpt, the next indication of he problem may be that the /var file system is filling up. This is a result of init creating an entry in /var/adm/wtmp file during each attempt to start the problem process. See below for a procedure to format the wtmp file in readable characters for diagnosis.


This technique makes use of the fwtmp utility which is part of the bos.acct fileset.

# lslpp -w /usr/sbin/acct/fwtmp

File                                    Fileset               Type

------------------------------------------------------------------

/usr/sbin/acct/fwtmp                    bos.acct              File



What's In The wtmp File


The actual content of the wtmp is not viewable, as the wtmp entries are written as binary structures (see /usr/include/utmp.h for the format). The fwtmp utility can be used to extract the contents into a human readable format.


For example we redirect the the contents of the /var/adm/wtmp file:


# /usr/sbin/acct/fwtmp < /var/adm/wtmp--> /tmp/wtmp_readable


At quick cat of the /tmp/wtmp_readable file and we find that the

file mainly consists of the following entries:

xvfb   xvfb   5 319596 0000 0000 1078170250    Mon Mar  1 11:44:10 2004

      xvfb   8 319596 0000 0001 1078170250    Mon Mar  1 11:44:10 2004

xvfb   xvfb   5 319598 0000 0000 1078170250    Mon Mar  1 11:44:10 2004

      xvfb   8 319598 0000 0001 1078170250    Mon Mar  1 11:44:10 2004

The first numeric column shows us the ut_type of entry, as defined in the utmp.h header file. The interesting types in our case are:


#define INIT_PROCESS    5    /* Process spawned by "init" */

#define LOGIN_PROCESS   6    /* A "getty" process waiting for login */

#define USER_PROCESS    7    /* A user process */

#define DEAD_PROCESS    8

In this example the "xvfb" entry is being started by init (signified by the "5" in column 3) and in the next line it's dying (ut_type = 8)


A quick check of the inittab file we find our problem:

# grep xvfb /etc/inittab

xvfb:2:respawn:/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &



In this case, the xvfb entry was starting an X terminal server daemon.


SOLUTION

The solution would then to resolve the command problem or change the entry in inittab from respawn to off using the chitab utility:

# chitab xvfb:2:off:'/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &'
In this specific case the trailing "&" sign was removed from the X Server command and it started up normally.
Historical Number
aix/tty/8


11 comments:

Unknown said...

Looking for training on IBM AIX, Check this site for trainings on emerging IT courses . Go here if you’re looking for training on IBM Aix.

Suseela said...


This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic.

Digital Marketing Company in Chennai

Digital Marketing Services in Chennai

dwarakesh said...


Ya that's great to have blog like this it may clear the many thought from this

Pega Training
PlSQL Training
AWS Training
Teradata Training
Qlikview Training
Informatica MDM Training

vinothika0895@gamil.com said...

Very interesting to read your blog. It make viewer to keep updated.

Selenium Training in Chennai

Unknown said...

Thanks for the information.
BIGDATATraining in Chennai

subha said...

This information is impressive; I am inspired by your post writing style & how continuously you describe this topic. After reading your post, thanks a lot.
C and C++ Training Institute in chennai | C and C++ Training Institute in anna nagar | C and C++ Training Institute in omr | C and C++ Training Institute in porur | C and C++ Training Institute in tambaram | C and C++ Training Institute in velachery

rocky said...

Thanks for the article. Is so amazing blog.
Python Training in Chennai

Python Training in Bangalore

Python Training in Hyderabad

Python Training in Coimbatore

Python Training

python online training

python flask training

python flask online training


Kanika said...

Good Post! , it was so good to read and useful to improve my knowledge as an updated one, keep blogging. After seeing your article I want to say that also a well-written article with some very good information which is very useful for the readers....thanks for sharing it and do share more posts like this.
Salesforce Online Training

Pushba said...

Good post thanks for sharing....
IELTS Coaching in chennai

German Classes in Chennai

GRE Coaching Classes in Chennai

TOEFL Coaching in Chennai

spoken english classes in chennai | Communication training

veera said...

Thanks for sharing article. The way you have stated everything above is quite awesome. Keep blogging like this. Thanks. | Certification | Cyber Security Online Training Course|

Ethical Hacking Training Course in Chennai | Certification | Ethical Hacking Online Training Course|

CCNA Training Course in Chennai | Certification | CCNA Online Training Course|

RPA Robotic Process Automation Training Course in Chennai | Certification | RPA Training Course Chennai|

SEO Training in Chennai | Certification | SEO Online Training Course


jdgvks said...

Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing....
oracle training in bangalore

hadoop training in bangalore
oracle training

oracle online training

oracle training in hyderabad

hadoop training in chennai

Labels

BlogCatalog