0516-070 : LVM system call found an unaccountable internal error.

On an IBM AIX 5.3 TL4 server,

# oslevel -s
/usr/bin/oslevel[601]: /tmp/sh158188.13: 0403-005 Cannot create the specified file.
# uname -a
AIX hostname 3 5 000EE4DC4C00

# # errpt -a
---------------------------------------------------------------------------
LABEL: DISK_ERR1
IDENTIFIER: 425BDD47

Date/Time: Mon Mar 24 10:40:07 EDT 2008
Sequence Number: 672
Machine Id: 000EE4DC4C00
Node Id: hostname
Class: H
Type: PERM
Resource Name: hdisk0
Resource Class: disk
Resource Type: scsd
Location: U0.1-P2/Z1-A8
VPD:
Manufacturer................IBM
Machine Type and Model......ST336607LC
FRU Number..................00P3068
ROS Level and ID............43353048
Serial Number...............000A91D5
EC Level....................H12094
Part Number.................00P2676
Device Specific.(Z0)........000003129F00013E
Device Specific.(Z1)........1217C511
Device Specific.(Z2)........0002
Device Specific.(Z3)........04132
Device Specific.(Z4)........0001
Device Specific.(Z5)........22
Device Specific.(Z6)........H12094

Description
DISK OPERATION ERROR

Probable Causes
MEDIA

User Causes
MEDIA DEFECTIVE

Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES

Failure Causes
MEDIA
DISK DRIVE

Recommended Actions
FOR REMOVABLE MEDIA, CHANGE MEDIA AND RETRY
PERFORM PROBLEM DETERMINATION PROCEDURES

Detail Data
PATH ID
0
SENSE DATA
0608 0000 0800 0000 0800 0000 0000 0000 0102 0000 F000 0300 0000 020A 0000 0000
1600 D280 0086 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 A15F 0006
#

# lsvg
rootvg
# lsvg -o
rootvg
#
rootvg
# lspv
hdisk0 000ee4dcf07f0297 rootvg
hdisk1 000ee4dcb0242e5d rootvg
#
# lqueryvg -Atp hdisk0
Max LVs: 256
PP Size: 26
Free PPs: 56
LV count: 12
PV count: 2
Total VGDAs: 3
Conc Allowed: 0
MAX PPs per PV 1016
MAX PVs: 32
Conc Autovaryo 0
Varied on Conc 0
Logical: 000ee4dc00004c00000000ff7751ed2b.1 hd5 1
000ee4dc00004c00000000ff7751ed2b.2 hd6 1
000ee4dc00004c00000000ff7751ed2b.3 hd8 1
000ee4dc00004c00000000ff7751ed2b.4 hd4 1
000ee4dc00004c00000000ff7751ed2b.5 hd2 1
000ee4dc00004c00000000ff7751ed2b.6 hd9var 1
000ee4dc00004c00000000ff7751ed2b.7 hd3 1
000ee4dc00004c00000000ff7751ed2b.8 hd1 1
000ee4dc00004c00000000ff7751ed2b.9 hd10opt 1
000ee4dc00004c00000000ff7751ed2b.20 lv10 1
000ee4dc00004c00000000ff7751ed2b.21 lv11 1
000ee4dc00004c00000000ff7751ed2b.22 upgradelv 1
Physical: 000ee4dcf07f0297 2 0
000ee4dcb0242e5d 1 0
Total PPs: 1084
LTG size: 128
HOT SPARE: 0
AUTO SYNC: 0
VG PERMISSION: 0
SNAPSHOT VG: 0
IS_PRIMARY VG: 0
PSNFSTPP: 4352
VARYON MODE: ???????
VG Type: 0
Max PPs: 32512
# lqueryvg -Atp hdisk1
Max LVs: 256
PP Size: 26
Free PPs: 56
LV count: 12
PV count: 2
Total VGDAs: 3
Conc Allowed: 0
MAX PPs per PV 1016
MAX PVs: 32
Conc Autovaryo 0
Varied on Conc 0
Logical: 000ee4dc00004c00000000ff7751ed2b.1 hd5 1
000ee4dc00004c00000000ff7751ed2b.2 hd6 1
000ee4dc00004c00000000ff7751ed2b.3 hd8 1
000ee4dc00004c00000000ff7751ed2b.4 hd4 1
000ee4dc00004c00000000ff7751ed2b.5 hd2 1
000ee4dc00004c00000000ff7751ed2b.6 hd9var 1
000ee4dc00004c00000000ff7751ed2b.7 hd3 1
000ee4dc00004c00000000ff7751ed2b.8 hd1 1
000ee4dc00004c00000000ff7751ed2b.9 hd10opt 1
000ee4dc00004c00000000ff7751ed2b.20 lv10 1
000ee4dc00004c00000000ff7751ed2b.21 lv11 1
000ee4dc00004c00000000ff7751ed2b.22 upgradelv 1
Physical: 000ee4dcf07f0297 2 0
000ee4dcb0242e5d 1 0
Total PPs: 1084
LTG size: 128
HOT SPARE: 0
AUTO SYNC: 0
VG PERMISSION: 0
SNAPSHOT VG: 0
IS_PRIMARY VG: 0
PSNFSTPP: 4352
VARYON MODE: ???????
VG Type: 0
Max PPs: 32512
#

hdisk0 and hdisk1 have the same LVs.

# synclvodm -Pv rootvg
0516-070 : LVM system call found an unaccountable
internal error.
0516-544 synclvodm: Unable to access volume group rootvg.
#
# redefinevg -d hdisk1 rootvg
# redefinevg -d hdisk0 rootvg
#
# lspv hdisk1
0516-070 : LVM system call found an unaccountable
internal error.
PHYSICAL VOLUME: hdisk1 VOLUME GROUP: rootvg
PV IDENTIFIER: 000ee4dcb0242e5d VG IDENTIFIER 000ee4dc00004c00000000ff7751ed2b
PV STATE: ???????
STALE PARTITIONS: ??????? ALLOCATABLE: ???????
PP SIZE: ??????? LOGICAL VOLUMES: ???????
TOTAL PPs: ??????? VG DESCRIPTORS: ???????
FREE PPs: ??????? HOT SPARE: ???????
USED PPs: ??????? MAX REQUEST: 256 kilobytes
FREE DISTRIBUTION: ???????
USED DISTRIBUTION: ???????
# lspv -l hdisk1
0516-070 : LVM system call found an unaccountable
internal error.
# lspv -l hdisk0
0516-070 : LVM system call found an unaccountable
internal error.
# lspv hdisk0
0516-070 : LVM system call found an unaccountable
internal error.
PHYSICAL VOLUME: hdisk0 VOLUME GROUP: rootvg
PV IDENTIFIER: 000ee4dcf07f0297 VG IDENTIFIER 000ee4dc00004c00000000ff7751ed2b
PV STATE: ???????
STALE PARTITIONS: ??????? ALLOCATABLE: ???????
PP SIZE: ??????? LOGICAL VOLUMES: ???????
TOTAL PPs: ??????? VG DESCRIPTORS: ???????
FREE PPs: ??????? HOT SPARE: ???????
USED PPs: ??????? MAX REQUEST: 256 kilobytes
FREE DISTRIBUTION: ???????
USED DISTRIBUTION: ???????
#


# unmirrorvg rootvg hdisk0
/usr/sbin/unmirrorvg[467]: /tmp/sh161360.13: 0403-005 Cannot create the specified file.
#


I will be booting this server into maintenance mode via NIM, then run fsck's,
logform, type exit, then try redefinevg and synclvodm again.....if it
still fails I will have to restore it.

Stay tuned.