Your Ad Here

IBM AIX/UNIX system storage administration ksh/perl scripting

Saturday, March 20, 2010

Tuesday, March 16, 2010

Command Respawning Too Rapidly From init, wtmp Filling Up

 Technote (FAQ)
How can I find out what command is respawning too rapidly and filling up my wtmp file?

A common situation is that the init process is repeatedly attempting to start a failing process. The init man page describes what happens when init finds an entry is being respawned:

If the init command finds that it is continuously running an entry in

the /etc/inittab file (more than five times in 225 seconds), it assumes

that an error in the entry command string exists. It then prints an

error message to the console and logs an error in the system error log.

After the message is sent, the entry does not run for 60 seconds. If

the error continues to occur, the command will respawn the entry only

five times every 240 seconds. The init command continues to assume an

error occurred until the command does not respond five times in the

interval, or until it receives a signal from a user. The init command

logs an error for only the first occurrence of the error.
To find out what is being respawned use the steps below.

1. Check the console or console logs

Check on the console to see if init is writing an error message similar to the one below:

0 Thu Jan 22 10:16:27 EST 2009

INIT: Command is respawning too rapidly. Check for possible errors.

id:  xvfb "/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &"

Or search through the console log using the alog command:
# alog -t console -o | more
2. Check errpt
Next there may be an entry in the errpt output with the label "INIT_RAPID" like below:



Date/Time:       Wed Jan 28 10:14:17 2009

Sequence Number: 1789

Machine Id:      00CC2F914C00

Node Id:         libgng

Class:           S

Type:            TEMP

Resource Name:   init



Probable Causes


User Causes


Recommended Actions


Detail Data


Command is respawning too rapidly. Check for possible errors.


id:  xvfb "/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &"

Both messages clearly identify the failing command that is being run out of the /etc/inittab file.

3. Check the wtmp file
If the warning messages are not noticed on the system console or in errpt, the next indication of he problem may be that the /var file system is filling up. This is a result of init creating an entry in /var/adm/wtmp file during each attempt to start the problem process. See below for a procedure to format the wtmp file in readable characters for diagnosis.

This technique makes use of the fwtmp utility which is part of the bos.acct fileset.

# lslpp -w /usr/sbin/acct/fwtmp

File                                    Fileset               Type


/usr/sbin/acct/fwtmp                    bos.acct              File

What's In The wtmp File

The actual content of the wtmp is not viewable, as the wtmp entries are written as binary structures (see /usr/include/utmp.h for the format). The fwtmp utility can be used to extract the contents into a human readable format.

For example we redirect the the contents of the /var/adm/wtmp file:

# /usr/sbin/acct/fwtmp < /var/adm/wtmp--> /tmp/wtmp_readable

At quick cat of the /tmp/wtmp_readable file and we find that the

file mainly consists of the following entries:

xvfb   xvfb   5 319596 0000 0000 1078170250    Mon Mar  1 11:44:10 2004

      xvfb   8 319596 0000 0001 1078170250    Mon Mar  1 11:44:10 2004

xvfb   xvfb   5 319598 0000 0000 1078170250    Mon Mar  1 11:44:10 2004

      xvfb   8 319598 0000 0001 1078170250    Mon Mar  1 11:44:10 2004

The first numeric column shows us the ut_type of entry, as defined in the utmp.h header file. The interesting types in our case are:

#define INIT_PROCESS    5    /* Process spawned by "init" */

#define LOGIN_PROCESS   6    /* A "getty" process waiting for login */

#define USER_PROCESS    7    /* A user process */

#define DEAD_PROCESS    8

In this example the "xvfb" entry is being started by init (signified by the "5" in column 3) and in the next line it's dying (ut_type = 8)

A quick check of the inittab file we find our problem:

# grep xvfb /etc/inittab

xvfb:2:respawn:/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &

In this case, the xvfb entry was starting an X terminal server daemon.


The solution would then to resolve the command problem or change the entry in inittab from respawn to off using the chitab utility:

# chitab xvfb:2:off:'/usr/bin/X11/X -force -vfb -x abx -x dbe -x GLX :1 &'
In this specific case the trailing "&" sign was removed from the X Server command and it started up normally.
Historical Number

How to Apply ACL to New Files Automatically

 Technote (FAQ)
How can I set up a directory that will automatically apply ACLs (Access Control Lists)?

Using an ACL is a way to set restrictions on the use of a file beyond the standard AIX or Unix discretionary access control. These can be set on an individual or group basis, and rights can be given or taken away as needed.
The concept of having a directory with ACLs on it, and then automatically applying those ACLs to any files created in that directory is called inheritance. AIX ACLs do not support inheritance, but NFSV4 ACLs do. NFSV4 ACL is platform-independent, so it can be supported on many clients or servers.

Most filesystem types in AIX do not support NFSV4 ACLs. CFS, UDF, JFS, and JFS2 with extended attribute version 1 (EAv1) do not support these. However JFS2 filesystems using EAv2 do, so they can be used for this purpose.

1. Use a JFS2 filesystem

If you have an existing JFS2 filesystem that can be used, or create one if you do not have one already

# crfs -v jfs2 -g datavg -m /myfs -A yes -a size=2G

2. Change the filesystem to use Extended Attributes Version 2

# chfs -a ea=v2 /myfs

3. Create a directory, or identify an existing directory

# cd /myfs
# mkdir newdir

4. Convert the directory to use NFS4 ACLs:

# aclconvert -t NFS4 newdir

5. Edit the ACL

# export EDITOR=/usr/bin/vi
# acledit newdir

You should see this in the file:

* ACL_type NFS4
* Owner: root
* Group: system
s:(OWNER@): a rwpRWxDaAdcCs
s:(OWNER@): d o
s:(GROUP@): a rRxadcs
s:(GROUP@): d wpWDACo
s:(EVERYONE@): a rRxadcs
s:(EVERYONE@): d wpWDACo

To allow inheritance for all files and directories underneath this dir, add the strings "fi" (for files) and "di" (for directories) to any ACLs you create. Those ACLs will be propagated to each file created from now on.

* ACL_type NFS4
* Owner: root
* Group: system
s:(OWNER@): a rwpRWxDaAdcCs fidi
s:(OWNER@): d o
s:(GROUP@): a rRxadcs
s:(GROUP@): d wpWDACo
s:(EVERYONE@): a rRxadcs
s:(EVERYONE@): d wpWDACo

Create a file in your directory and check the ACL list on it:

# cd newdir
# touch newfile
# aclget newfile

* ACL_type NFS4
* Owner: root
* Group: system
s:(OWNER@): a rwpRWxDaAdcCs fidi

How to use netgroups with NFS without NIS

Technote (FAQ)


How to use netgroups with NFS without NIS?


1. Create /etc/netgroup file on NFS server

Here's an example of a netgroup file that contains a group called "group1", which consists of three hosts:

# cat /etc/netgroup

group1 (tcplabbso,,) (bonesbso,,) (srcmstrbso,,)

For more specific information about the syntax of a netgroup file, please see the man page below:

2. Modify /etc/exports file (or use smitty) to specify access for netgroup, i.e.:

# cat /etc/exports

/tmp -sec=sys:krb5p:krb5i:krb5:dh,access=group1

3. Unexport and re-export

# exportfs -ua

# exportfs -av

4. Test

Monday, March 15, 2010

mkvdev creating SEA adapter generates: 0514-040 Error initializing a device into the Kernel.

 Technote (FAQ)
Why am I getting this error when I am running my mkvdev command to create an SEA:

Method error (/usr/lib/methods/cfgsea):

0514-040 Error initializing a device into the kernel.
There are many things that can cause this error when creating a Shared Ethernet Adapter. The main idea when troubleshooting the problem is to find out if something is locking down the underlying adapters that is keeping them from being able to be used in the SEA. Here are some common things to check:
1) #netstat -in --> do you see any en or et network interfaces for any of the underlying adapters? If so you will need to detach them from the system, and it is recommended to remove them so that they do not come back on a reboot. Sample commands: where X is the number of the adapter

#ifconfig enX down detach

#rmdev -dl enX

2)You will want to make sure all the ent adapters that are involved in the command are showing up as "available". You can run this command to view the adapters and their state:

#lsdev -Cc adapter |grep ent

3) Make sure that the virtual adapter that is being used as the "trunk" adapter, i.e. the adapter that follows the -vadapter attribute, has the "access external networks" section checked in the radio box. See below screen shot for example:

4) Verify that the ent adapter is not being used by another device such as a etherchannel or another SEA adapter. The only way to verify this is to check out the attributes of all the other SEA, and etherchannel adapters. You can do this by running the following command in the oem_setup_env: (where X is the number of the SEA or etherchannel you want to view)

#lsattr -El entX (where X = the number of the adapter)

5) Another way to cause this error is if the physical adapter and the virtual adapter "check_sum_offload" setting are not the same. They must either both be yes or both be no. The command to check this attribute is:

#lsattr -El entX (where X = the number of the adapter. Run this on both the physical and virtual adapter to be used in the SEA)

6) If you are using an HEA adapter you will want to verify that your vio server is the promiscuous lpar for that physical port. See DCF document: "HEA / IVE Host Ethernet Adapter / Integrated Virtual Ethernet, Setting the Promiscuous Lpar ".,promiscuous

7) Another way to get this error with an HEA is if the VIO server partition does not have enough physical memory. The only way to determine if you need more physical memory to get past this error is to allocate some more memory to the lpar and re-activate. Once this is done, re-try your mkvdev command.

8) This is not a total comprehensive listing as there may not be any such thing. This error message is very generic and new problems arise that can cause this message to be generated. The most important thing to keep in mind when getting this error is that for some reason the adapters involved in creating the SEA are unable to be used for one reason or another. Troubleshooting should be narrowed down to examine each adapter to find out what the problem may be.

NOTE: All commands in this document are to be run from the root prompt in "oem_setup_env" with the exception of the mkvdev command itself. The mkvdev command needs to be run in the CLI environment.

    NIM installation and backup of the VIO server

     Technote (FAQ)
    How do I backup and install VIOS using NIM?



    This guide is intended for those who are new to installing the VIO server using NIM.

    For a more advanced, “official,” or higher level NIM specific guide you can check out the recently released redbooks :
    NIM From A-Z in AIX 5L
    Just go to and type in SG24-7296-00 in the search field on the upper right hand side of the page.
    You can type SG24-7940-03 in the search field for the VIOS redbook

    I will try to keep this as reader friendly as possible. Feel free to make suggestions on how to improve this document, and if there are any procedures you would like to see added in the future, let me know and I’ll do my best to add these procedures.

    Finally, there always will be someone to tell you how you should appropriately type out “commands to be executed.” Some people prefer that commands are put in single quotes, others prefer that they’re in italics.....whenever there is an actual command to execute, the example of that command will look just like your command line interface on your AIX system.

    For example if I want you to do a listing of files in a directory I would have the example of :

    # ls -al

    I prefer to do it this way because this is exactly what you’ll see when you type it on your system....(and ok, it’s also easier to cut and paste the commands from my test system into this document once I’ve verified they work correctly).

    What this guide does and does not cover :

    Does :
    - Key words and definitions
    - Setup of the client definition on the NIM master
    - Setup of NIM resources

         Copy the VIOS mksysb image from the CD to your NIM master



         'bosinst_data' resource
    - Installation of VIOS to NIM clients
    - Backup of the VIO server using the "nim_resrouces.tar" package
    - Frequently Asked Questions

    --SMIT and Command line processes for most of the operations I cover

    Does not:
    - Any websm interface procedures on how to use NIM. Websm is not used enough to spend any time going into separate sections on how to use it.
    - NIM communication information (including nimsh and firewall issues)

    Key words and definitions :

    If you are unfamiliar with NIM I highly recommend reading through this section and also use it as a reference while reading through this guide. If you’re figuring, “eh he’ll explain all this later so I’ll just skip this part,” you will be flipping back to this.

    So this is sort of like a “reverse-glossary”. I’ll give the best and easiest to understand description of these terms so that you’ll hopefully have a much easier time understanding any new concepts you’re unfamiliar with. In all cases - actually using the files/keywords are handled in greater detail in their corresponding “How To...” sections.

    Important Files and Directories:

    - /etc/bootptab :
    This file will exist on the NIM master. In a quiet NIM environment with no operations that require a client to boot, this file will be empty (except for the pre-existing commented section). This file gets updated automatically by the NIM master when a NIM operation is executed that requires the client machine (from here on out referred to as the "VIO server") to boot from a NIM SPOT. If this file contains incorrect information about either the master or the VIO server, the boot operation will fail. While this file “can” be edited manually to fix a bootp issue - it should not be, as you are only applying a “band-aid” fix to an existing issue in your NIM environment....but, sometimes it’s 5pm on a Friday and you’re ready to go home, right ?
    (Also note related entry ‘bootp’)

    - /etc/exports :
    This is not a “NIM specific” file, it is a NIM critical file. Any sort of installation, boot, mksysb, savevg....etc operation requires the use of NFS. This file will be updated with which locations are NFS exported from the master to the VIO server and the permissions associated with those exports. If these entries are incorrect or incomplete you will run into boot failures, permission problems, and other errors commonly associated with NFS. This is a text file and also “can” be edited manually to sometimes “band-aid” a problem, but should only be done so with care in knowing exactly what you’re doing. The good thing is, if we mess up this file we can remove it and recycle NFS. The file can be recreated.

    - /etc/hosts :
    While not a “NIM specific” file, it is also a NIM critical file. This file is sort of like a phone book. It gives a relationship between a system’s hostname and an IP address. Much like a telephone, if you dial the wrong number you get the wrong person. In NIM, if your ip address does not match up to the correct hostname, your install fails. This is a text file and can be edited manually. There should also only be 1 entry per ip/hostname. I personally prefer to make sure my NIM master has all entries in the /etc/hosts file and are of the following format :
    <ipadress> <shortname> <longname>

    - /etc/niminfo:
    This file should always exist on the NIM master. This file is built when you first initialize the NIM environment. This is simply a text file so feel free to ‘cat’ or ‘more’ the file and look at the entries included in there. You do not want to manually edit this file if there is a mistake in the definition of the master. In this case you will want to redefine the master, or use the feature in NIM to change the master’s attributes (hostname, gateway....etc).

    - /tftpboot :
    This directory holds the boot images, along with informational files about the VIO server that is being installed by the NIM master. The file names that are generated generally look like this: à This is the boot image that will be used to boot the VIO server.
    The format for the file is:
    A <VIOserver_hostname> file is created and linked to the boot image on the NIM master
    Example: (This is linked to the boot image. Example:
    A VIO server info file will also be created and is named: <VIOserver_hostname>.info

    Important Keywords :

    Allocate/Allocation :
    This process is what allows your VIO server to access resources in NIM. The master uses NFS to perform the allocation process. You can see which resource types are allocated to your VIO server by using the following command :
    # lsnim -a spot
    spot = VIOS-A_spot
    spot = VIOS-B_spot
    spot = VIOS-C_spot

    In this example environment we have 3 VIO servers defined as NIM clients. Since the SPOT resources are created from the mksysb image of a VIO server, giving the SPOT resource a descriptive name that relates back to the hostname of your VIO server is helpful.

    So we can see that ‘VIOS-A’ has a SPOT called “VIOS-A_spot”, ‘VIOS-B’ has a SPOT called “VIOS-B_spot, and ‘VIOS-C’ has a SPOT called “VIOS-C_spot”.

    Base Operating System Installation :
    Also commonly called (and referred to from here on out) as a bos_inst operation. This simply refers to the fact that you are initiating a boot and installation to a VIO server.

    Bootp :
    This is the initial communication made between the NIM master and the VIO server during a boot or bosinst operation. In order for this to be successful several factors must be met :
    1. - bootpd must be running on the NIM master
    2. -You must have the correct IP information for the NIM master and target VIO server
    3. - the /etc/bootptab must be populated correctly
    4. - If the master and VIO server are on separate networks, the router must be set to forward bootp packets.
    There are other causes of failure, but checking/verifying those 4 will solve most bootp issues.

    Tftp (Trivial File Transfer Protocol):
    When the VIO server has been rebooted for a bos_inst operation you don’t have access to normal TCP communication. Once bootp connection has successfully been achieved, the NIM master uses tftp to transfer over the <clientname>.info file and the boot image to the VIO server.

    if1= : Or interface 1
    This is known as the ‘nim network’. Every machine, even the master, is placed on a defined NIM network. A machine who has multiple adapters defined to NIM will have “if2=” and “if3=”....etc attributes. When your NIM master is initialized, you will create a network name for the master and VIO server on the same subnet as the master. If you name this network “master_net” for example, then all VIO server NIM clients on the same subnet as the master will have their “if1=” line set to “master_net”. If you add additional VIO server NIM clients that are on separate subnets, then you will need to create new network names. You can see the “if1=” information from an :
    # lsnim -l master |more
    # lsnim -l <client>

    You can get further information about the network name by running an ‘lsnim -l <network_name>’.
    Having incorrect networking information is probably the leading cause of NIM installation failures. This attribute and the information used when creating networks is extremely important to make sure you have correct.

    Client (NIM client) :
    A standalone VIO server, or lpar in a NIM environment, other than the NIM master. Clients (VIO servers) use the resources that reside on the NIM master to be installed.

    Master (NIM master) :
    The only system in a NIM environment that has permission to run NIM commands remotely on other NIM clients (VIO servers). A NIM client (VIO server) can only have one NIM master. The NIM master must be at a level equal to or higher than the AIX level that is being installed (and accompanying VIOS software) to the VIO server.

    The NIM master will also hold all of the NIM resources. Due to this fact, you will want to make sure that the NIM master has plenty of space available for storing these resources. It is ideal to have a separate volume group (example: nimvg) so that the rootvg does not get out of control in size.

    Resource (NIM resources) SPOT:
    The SPOT resource (stands for Shared Product Object Tree in case you were wondering) is responsible for the following :
    - Creating a boot image to send to the VIO server over the network.
    - Running the commands needed to install the VIO server.

    You will need to create a SPOT from your NIM VIOS mksysb resource. The VIOS mksysb contains all of the filesets and commands needed to create the SPOT. A SPOT created from an AIX lpp_source is not a valid SPOT to use when installing a VIO server. Essentially the SPOT is a /usr filesystem just like the one on your NIM master. You can think of it as having multiple “mini-systems” on your NIM master, because each SPOT is its own /usr filesystem. Just like your NIM master’s /usr filesystem, going in there manually and messing around with files can easily corrupt it. The good thing about a SPOT however, is that it is easily rebuilt.

    Resource (nim resources) mksysb :
    This is simply a a mksysb image of the VIO server. The mksysb image resides on the NIM master, and the resource can be defined in one of two ways:
    -From an existing mksysb image residing on the NIM master already (extracted from a nim_resources.tar package).
    -From the mksysb image copied from the VIOS installation media on to hard disk (this is described in more detail later in this document).

    You can also back up an already existing VIO Server and a nim_resources.tar package is created. The contents of this nim_resources.tar package can be stored on the NIM master for installations to new VIO servers, and also for the recovery of VIO servers using NIM. The nim_resources.tar file contains all the necessary resources to restore the Virtual I/O server. This includes the mksysb image, the file, the network boot image, and SPOT resource. When the VIO server is backed up, the nim_resources.tar package will be stored on an NFS mounted file system. More detailed steps for creating the nim_resources.tar file with the ‘backupios’ command will be given later in this document.
    Resource (nim resources) bosinst_data:
    When booting from an installation image to install a system, the system boots to what are known as the “BOS Menus” or Base Operating System Installation Menus. Here is where you select your console, the language to use during the installation, and what disks to install to amongst other things. When using NIM, a “bosinst_data” resource can be created, and it will answer all of the questions that are asked by the “BOS Menus” automatically. This resource is used when installing to a NIM client that may be in another building, down the road or half way across the country. This way, once you kick off the installation from the NIM master, there is no further interaction that is required. The system should (ideally) install and reboot itself afterwards.

    Backupios (VIO server backup):
    The ‘backupios’ command will back up the Virtual I/O Server and user-defined virtual devices to tape, DVD, or a remote file system.You can back up the Virtual I/O Server and user-defined virtual devices using the backupios command.
    A mksysb (as discussed above) has a “built in” file. If the option in that file
    (PROMPT =) is set to yes, this file really does nothing as the choices you make in the BOS menus will override the options in the file. However, if the mksysb was created to have that option set to no, then we can create a new bosinst_data resource which will trump the one that is part of the mksysb.

    Define the VIOS partition as a NIM client:

    From command line :

    # nim -o define -t standalone -a platform=chrp -a if1="find_net lucidbso 0" -a netboot_kernel=mp -a connect=shell <clientname>

    Breaking down the command
    # nim -o define -t standalone : This is the NIM operation of defining an object of type=standalone.

    ... -a platform=chrp : The platform type of the target system.

    ...-a if1=”find_net lucidbso 0’ : Used when a client is part of an existing NIM network. In this case my NIM client belongs to my master’s NIM network (i.e. They’re on the same subnet) so I don’t need to give it further information. If this client needed to be on a new network then we would not use the “find_net” but rather the “net_definition=<value>.......” options where you’d need to specify “networktype” “subnetmask” “clientgateway” “networkname” attributes.

    ... -a netboot_kernel=mp :This should always be set to mp for a multiprocessor and in the future they will probably have an mp64 option.

    ... -a connect=shell : This will be the connection type used. You can either choose shell (rsh) or nimsh.

    ... <clientname> : The clients resolvable hostname.

    From SMIT :

    # smitty nim_mkmac
    # smitty nim
    ==> Administration Tasks ==> Manage Machines ==> Define a Machine

    From here you are asked what the hostname of the NIM client (VIO server) is. If the master can resolve the client’s hostname and he already has a NIM network associated with the client’s network then it will “pre-generate” a final acceptance screen for you (see example 1). If the master does not already have a NIM network defined for this client, then you will be prompted for more information and be asked to create a new network for this new client (see example 2).

    * NIM Machine Name [clientname]
    * Machine Type [standalone]
    * Hardware Platform Type [chrp] Make sure this is set to ‘chrp’
    Kernel to use for Network Boot [mp]
    Communication Protocol used by client []
    Primary Network Install Interface
    * Cable Type bnc
    Network Speed Setting []
    Network Duplex Setting []
    * NIM Network master_net
    * Host Name client_hostname
    Network Adapter Hardware Address [0]
    Network Adapter Logical Device Name []
    IPL ROM Emulation Device []
    CPU Id []
    Machine Group []
    Comments []

    Notice that you actually don’t have to fill in anything else as a required field. The master can resolve the VIO server's hostname, and he has put it on the same network (master_net) because in this case, the client and master are on the same network.
    I recommend leaving the optional fields blank as they currently
    Pressing <enter> from here will define the client.

    * NIM Machine Name [clientname]
    * Machine Type [standalone]
    * Hardware Platform Type [chrp]
    Kernel to use for Network Boot [mp]
    Communication Protocol used by client []
    Primary Network Install Interface
    * Cable Type bnc
    Network Speed Setting []
    Network Duplex Setting []
    * NIM Network [10_14_20_Net]
    * Network Type ent
    * Ethernet Type Standard
    * Subnetmask []
    * Default Gateway Used by Machine []
    * Default Gateway Used by Master []
    * Host Name clientname
    Network Adapter Hardware Address [0]
    Network Adapter Logical Device Name []
    IPL ROM Emulation Device []
    CPU Id []
    Machine Group []
    Comments []

    Notice that there is much more to fill out here, because we’re defining a new network within NIM. We’ll say my VIO server's ip = I’m naming my NIM network 10_14_20_Net so that it is descriptive just by looking at the name. Also, any other VIO server on this same network will now be auto-defined on this new network and the SMIT screens will look like example 1 above. We’ll also need to enter my subnetmask and the gateway used by this client. The master’s gateway is filled in for you. Once this successfully completes I‘ll now have 2 NIM networks.......master_net and 10_14_20_Net.

    Setup of NIM resources:

    Now that the client machine has been defined, you are ready to set up the resources needed for the VIOS install using NIM.

    Copy the VIOS mksysb image from the CD to your NIM master:

    Mount the VIOS base CD and copy the VIOS mksysb image from the CD (in /usr/sys/inst.images) to your NIM master:
    # mount -o ro -v cdrfs /dev/cd0 /mnt
    # cd /mnt/usr/sys/inst.images
    # cp mksysb_image /export/mksysb/mksysb_image
    If using VIOS 1.5 or higher media, the mksysb file may be split into two parts. To combine these two parts and copy them to hdisk, run the following:
    # cat /mnt/usr/sys/inst.images/mksysb_image /mnt/usr/sys/inst.images/mksysb_image2 > /dir/filename

    ***you can substitute any path you would like to save the combined mksysb image, for ‘/export/mksysb_image

    Define the mksysb resource on the NIM master:

    # nim -o define -t mksysb -a server=master -a location=/export/mksysb/mksysb_image
    # smitty nim_mkres => mksysb =>
    Define a Resource

    * Resource Name [VIOS_1_5] Give the resource a name
    * Resource Type mksysb
    * Server of Resource [master] + Chose your NIM master
    * Location of Resource [/export/mksysb/mksysb_image] / The full path of the mksysb file copied to hdisk

    àLeave all other entries set to their default values

    Define a SPOT from the mksysb resource:

    # nim -o define -t spot -a server=master -a location=/export/spot –a source= VIOS_1_5 VIOS_1_5_SPOT


    # smitty nim_mkres => spot =>
    Define a Resource
    * Resource Name [VIOS_1_5_spot] give the spot a name
    * Resource Type spot
    * Server of Resource [master] + chose your NIM master
    * Source of Install Images [VIOS_1_5] + select the mksysb resource
    * Location of Resource [/export/spot] / location to store the SPOT

    àLeave all other entries set to their default values

    Define the 'bosinst_data' resource on the NIM master:

    You'll also want to copy the file from the CD media to the NIM master and define that as a resource:
    (If the CD has been mounted as described earlier, run the following command. If the CD was mounted in a different way, then simply change directories to the mount point and copy the file)
    # cd /tmp
    # mkdir VIOS_bosinst
    # cp /mnt/ /tmp/VIOS_bosinst
    # nim -o define -t bosinst_data -a server=master –a location=/tmp/VIOS_bosinst VIOS_1_5_bosinst_data


    # smitty nim_mkres => bosinst_data =>
    Define a Resource
    * Resource Name [VIOS_1_5_bosinst] Give the resource a name
    * Resource Type bosinst_data
    * Server of Resource [master] + chose the NIM master
    * Location of Resource [/tmp/VIOS_bosinst/] / location of the file copied from CD

    àLeave all other entries set to their default values

    Perform the bos_inst operation to set up the NIM resources:

    for the installation:
    # nim –o bos_inst –a source=mksysb –a spot=<spotname> -a accept_licenses=yes –a mksysb=<mksysbname>

    # smitty nim_bosinst => select the nim client (VIOS lpar) from the list => select ‘mksysb’ as the installation type => select the VIOS mksysb from the list of mksysbs => select the spot created from the VIOS mksysb =>
    Install the Base Operating System on Standalone Clients

    * Installation Target VIOS-A
    * Installation TYPE mksysb
    * SPOT VIOS_1_5_spot
    LPP_SOURCE [] +

    BOSINST_DATA to use during installation [VIOS_1_5_bosinst] + Select the bosinst_data resource created from the VIOS mksysb

    ACCEPT new license agreements? [yes] + Set to ‘no’ by default so you must change to ‘yes’

    Initiate reboot and installation now? [yes] + Set to ‘no’ for the pull installation

    àLeave all other entries set to their default values

    7. The master is set what ?

    The NIM master is now ready and waiting for a bootp request from the VIO server. In order to do this we need to boot the VIO server into SMS (System Management Services). Depending on your machine type there are a few different ways to boot to SMS.

    1. Using a graphical terminal, during system boot you’ll see icons going across the bottom of the screen. Anytime after the “keyboard” icon and before the “speaker” icon you can press the F1 key to indicate that you wish to boot to SMS.

    2. Using an ASCII terminal, during system boot you’ll see words come across the bottom of the screen. Anytime after the word “keyboard” and before the word “speaker” you can press the 1 key to indicate you wish to boot to SMS. This will be the 1 key on the number line, not on the number pad.

    3. If you have an HMC managed environment, you can simply boot your VIOS server using the SMS profile.

    If you are one of those whose terminal seems to not come up in time to be able to see the icons or words come across the screen, the LED display during the time you have to press the appropriate button is E1F1. Also, sometimes you’ll notice on graphical systems it specifically asks for 1 instead of should display on the screen the correct button to press.

    Your firmware level can depend on what sort of options you see in SMS. I don’t have access to all variations of the menu options, so I’ll just use the one my NIM master has. You should be able to navigate closely enough, as the wording should be similar.


    1. Select Language

    2. Change Password Options

    3. View Error Log

    4. Setup Remote IPL (RIPL (Remote Initial Program Load))

    5. Change SCSI Settings

    6. Select Console

    7. Select Boot Options

    First of all we need to provide the IP addresses necessary to tell the VIO server who he is, and where to boot from.
    4. Setup Remote IPL (RIPL (Remote Initial Program Load))

    Next you’ll have a selection of adapters to use. Select your adapter that corresponds to the adapter/host you defined in NIM. You will not see “ent0/ent1...etc” options. You will however see the hardware addresses and slots.

    We then are brought to 3 options :

    1. IP Parameters

    2. Adapter Configuration

    3. Ping Test

    Chose the option to set the IP Parameters

    Now we need to set our IP Parameters.

    1. Client IP Address [###.###.###.###]

    2. Server IP Address [###.###.###.###]

    3. Gateway IP Address [###.###.###.###]

    4. Subnet Mask [###.###.###.###]

    Enter the correct addresses for each field.


    If your NIM master and client are on the SAME subnet, then you will set the Gateway IP Address to be the NIM master’s ip address. There were a few firmware levels that made you use if the master and client were on the same network, but running into that is very rare. Use the NIM master’s IP.

    If you are unsure or need to verify if your master and client are on the same network you can always go back to your NIM master and check to see what NIM thinks. Of course, NIM is only correct if the information provided to it was correct originally.

    On the NIM master you’ll want to compare the “if1=” lines on the master and client :

    # lsnim -l master |grep if1

    if1 = master_net shadoebso

    #lsnim -l VIOS-A |grep if1

    if1 = master_net dipperbso

    So we know the master and VIO server are defined on the same subnet. If the VIO server is defined on a different NIM network than the master, then use the VIO server’s gateway as the Gateway IP Address entry.

    After setting the IP Addresses use ‘M’ to return to the main menu. You typically do not want to go into the “Adapter Parameters” (option 2 on the previous screen) to change the adapter parameters or disable spanning tree. Spanning Tree goes out and wakes up parts of the network that might have been disabled since they’re not in use. This can increase performance but also block bootp if it is not expecting to receive a request through that portion of the network.

    Also, with the Ping Test....the ping test is not a reliable way to determine if bootp is going to work. In fact, we use it the other way around. If bootp is failing for some reason we may come back and check the ping test to see if that is having a problem as well. Do not presume bootp will fail if the ping test fails. It may, but it is not a reliable indication.

    With our IP Parameters set we should now be back at the main menu.


    1. Select Language

    2. Change Password Options

    3. View Error Log

    4. Setup Remote IPL (RIPL (Remote Initial Program Load))

    5. Change SCSI Settings

    6. Select Console

    7. Select Boot Options

    Now we’re ready to select our boot device.
    7. Select Boot Options

    The next menu should come up :

    1. Select Install or Boot Device

    2. Configure Boot Device Order

    3. Multiboot Startup

    You can take either option 1 which will make this a 1 time boot device selection, or option 2 which will permanently change the boot device order until either you or the system changes it back to your boot drive after the install is complete.
    1. Select Install or Boot Device

    Select Device Type :

    1. Diskette

    2. Tape

    3. CD/DVD

    4. IDE

    5. Hard Drive

    6. Network

    7. List all Devices

    Trust me, make your life easier and select :
    7. List all Devices

    The system will go out for you and scan itself to determine which devices are available to boot from. All of your available boot devices will be displayed here. The menu can be a little tricky here. If you have a device pre-selected it will have a 1 next to it under the “Current Position” column. Use the “Select Device Number” listing to choose the device you want to boot from.

    The next screen will offer you three choices :

    1. Information

    2. Normal Mode Boot

    3. Service Mode Boot

    Select :
    2. Normal Mode Boot

    It shouldn’t really matter if you select normal or service mode, I always select normal mode.

    Finally it asks if you’re sure you want to exit from SMS. Select ‘yes’ and let the boot go.

    What you SHOULD see :

    You’ll likely see a brief splash screen then the bootp request attempt.

    Ideally you’ll see something like :

    BOOTP : S=1 R=1

    Which indicates it sent and received a request successfully.

    Next the master will tftp the boot image over so you should see a number rapidly increasing to roughly 24000-ish. Once that is complete the VIO server will go through all of the necessary LED codes to establish proper communication and NFS mounts. After that you will be presented with the BOS menu screens.

    Backing up the VIO server:

    Creating a "nim_resources.tar" file using 'backupios'

    To backup the VIO server to a remote file system:
    1. Create a mount directory where the backup image, nim_resources.tar package, will be written to.
    Example: # mkdir /VIOS-A/backup
    2. Mount an exported directory from the NIM master on the mount directory
    Example: # mount server1:/export/VIOS_backup /VIOS-A/backup
    3. Run the backupios command with the –file option. Make sure to specify the path to the mounted directory
    Example: # backupios –file /VIOS-A/backup
    This will create the nim_resources.tar package


    1. Can an lpp_source and SPOT created from AIX media be used to install the VIO server?
    No. The AIX media and the VIOS media are not interchangeable. All of the device drivers and the other software needed to install the VIO server are located on the VIOS media. The AIX media does not contain all of the same device drivers or software necessary to install the VIO server, so AIX resources cannot be used to install a VIO server.

    2. Do I need to use an lpp_source to install the VIO server using NIM?
    No. All of the device drivers and software packages needed to install the VIO server are in the VIO server mksysb.

    3. At what level does the VIO server media start containing two mksysb images that must be combined to create the NIM mksysb resource?
    If using VIO server media version 1.5.1 or higher, there will be two mksysb images that must be combined to create the NIM mksysb resource.

      Wednesday, March 10, 2010

      Backupios Fails with 0512-008 savevg

      Backupios command fails with 0512-008 savevg: The mkvgdata command failed. Backup canceled.
      $ backupios -file /home/padmin/mksysb/ibm74vioa_mksysb -mksysb
      /home/padmin/mksysb/ibm74vioa_mksysb doesn't exist.

      Creating /home/padmin/mksysb/ibm74vioa_mksysb
      Backup in progress. This command can take a considerable amount of time
      to complete, please be patient...

      Creating information file (/ for rootvg.
      0512-008 savevg: The mkvgdata command failed. Backup canceled.

      /usr/bin/mkvgdata[1068]: -: more tokens expected

      This error is caused by having a user created filesystem mounted within rootvg.

      Resolving the problem
      Unmount all user-created filesystems in rootvg and re-run backupios command.
      Note: crfs command is NOT part of the padmin restricted user, and therefore it is NOT supported to create user filesystems in the Virtual I/O Server.


      Atape trace log files are filling my /var filesystem, how do I disable Atape rmt tracing?


      Atape trace log files are filling my /var filesystem, how do I disable Atape rmt tracing?


      Atape trace_logging attribute is "off" by default, but may have been left "on" from prior troubleshooting.


      The Atape rmt device attributes include a "trace_logging" attribute that will continuously write "Atape.rmt<#>.trace<#>" files to the /var/adm/ras directory if enabled.
      You can disable Atape trace logging by changing the trace_logging attribute to "no" with chdev. For example:

      chdev -l rmt0 -a trace_logging=no

      IVM CLI Examples to set HEA Attributes on Power Systems (p6)

      What does the command line interface (CLI) syntax look like for setting host Ethernet adapter (HEA) attributes?

      VIO Server 1.5 and 2.x can run on entry level p6 Power System servers which can be managed using the Integrated Virtualization Manager (IVM) graphical and command line interfaces. Most of the graphical functions are intuitive, but it helps to have examples of some commonly requested syntax for configuring the HEA adapter using IVM CLI.


      Following are some examples of setting the most command HEA attributes using IVM CLI interface.

      - Setting promiscuous LPAR option on a HEA port.
      Example sets the promisc attribute on port 1 of group 1 to LPAR ID 1 (the VIO server).

      chhwres -r hea -o s -l 23000000 -g 1 -a promisc_lpar_id=1 --physport 1

      - Setting the MES value on an HEA adapter (requires system to be power cycled)
      Example sets the MES value for port group 1 to value of 1 (default value is 4).

      chhwres -r hea -l 23000000 -g 1 -o s -a pend_port_group_mcs_value=1

      The 23000000 number in above commands is the adapter ID. The adapter ID for Power System models can differ. Use the lshwres command to view the adapter ID as well as other attributes of the HEA device.

      lshwres -r hea --rsubtype phys --level port

      lshwres -r hea --rsubtype phys --level port_group

      You can run lshwres or chhwres on the IVM command line to get usage syntax. You should also be able to view the man pages or view the command descriptions documented in following reference.

      Virtual I/O Server and Integrated Virtualization Manager command descriptions

      Why does the FTP option fail in my updates for HMC 7.3.4

      Why does the FTP option fail in my updates for HMC 7.3.4

      HMC update modifies feature of the HMC update using the remote server. 

      Monday, March 08, 2010

      NPIV FAQs for Vritual I/O Server Environment


      This document covers the most frequently asked questions for NPIV in a Virtual I/O Server (VIOS) environment.
      This applies to VIOS version 2.1 and above.


      1. What is NPIV?
      N_Port ID Virtualization(NPIV) is a standardized method for virtualizing a physical fibre channel port. An NPIV-capable fibre channel HBA can have multiple N_Ports, each with a unique identity. NPIV coupled with the Virtual I/O Server (VIOS) adapter sharing capabilities allow a physical fibre channel HBA to be shared across multiple guest operating systems. The PowerVM implementation of NPIV enables POWER logical partitions (LPARs) to have virtual fibre channel HBAs, each with a dedicated world wide port name (WWPN). Each virtual fibre channel HBA has a unique SAN identity similar to that of a dedicated physical HBA.

      2. What are the minimum requirements for NPIV?

    • A POWER6-based System p server OR

    • Minimum system firmware levels

      • EL340_039 for IBM Power 520 and 550
      • EM340_036 for IBM Power 560 and 570

    • One of the following JS blade types. Note: NPIV is only supported on the BladeCenter H

      • JS12 (7998-60X)
      • JS22 (7998-61X)
      • JS23 (7778-23X)
      • JS43 (7778-23X + FC8446)
      • BladeCenter H (7989-BCH)

    • Minimum one supported Fibre Channel adapter (see FAQ 3 & 4)

    • NPIV-enabled SAN switch.
      Only the SAN switch which is attached to the Fibre Channel adapter in the VIOS needs to be NPIV-capable. Other switches in your SAN do not need to be NPIV-capable.

    • Switch levels

      • Brocade v6.1.0 or later
      • McData v9.7 or later
      • Cisco v3.2 (3) or later


      • HMC 7.3.4 or later
      • Virtual I/O Server (Fixpack 20.1) or later
      • AIX 5.3 TL 9 or later
      • AIX 6.1 TL 2 or later
      • IBM i 6.1 with 6.1.1 LIC or later

      3. What are the supported Fibre Channel adapters for NPIV in a POWER6 System p server?
      The 8 Gigabit Dual Port Fibre Channel adapter, feature code 5735, is the only supported adapter.
      The minimum firmware requirement to enable NPIV for AIX on this adapter is 110305 (# lsmcode -d fcs#).
      You can obtain this image from the Microcode downloads site--select Power from Product Group, then Firmware and HMC Product.

      4. What are the supported Fibre Channel adapters for the IBM BladeCenter H?
      Fibre Channal Module Feature Code
      ---------------------------------------------- ----------------
      Emulex 8Gb Fibre Channel Expansion Card (CIOv) 8240
      QLogic 8Gb Fibre Channel Expansion Card (CIOv) 8242 (DUAL port)
      QLogic 8Gb Fibre Channel Expansion Card (CFFh) 8271

      Note: NPIV is only supported on the BladeCenter H and the firmware level of the Fibre Channel adapter must support NPIV.

      5. Can I have dual NPIV capable Fibre Channel adapters on the same VIOS in different zones/fabrics?

      6. Is it possible to SAN boot the VIOS using the same Fibre Channel adapter being used to service NPIV traffic?
      With proper zoning, it is possible to SAN boot the VIOS with the same host bus adapter that is servicing the NPIV traffic. The VIOS would need to be running before the NPIV-based traffic can serve the clients.
      Use care with this configuration because even a simple error in zoning could disable a VIOS and result in loss of the services it provides.