Recovering a broken mirror in Tru64 (Kilala.nl (Cailin Coilleach)) by Cailin Coilleach

Today I faced the task of replacing a failing hard drive in one of our Tru64 boxen. The disk was part of a disk group being used to serve plain data (as opposed to being part of the boot mirror / rootdg), so the replacement should be rather simple.

After some poking about I came to the following procedure. Those in the know will recognize that it's very similar to how Veritas Volume Manager (VXVM) handles things. This is because Tru64 LSM is based on VXVM v2.

* voldiskadm -> option 4 -> list -> select the failing disk, this'll be used as $vmdisk below.
* voldisk list -> select the failing disk, this'll be used as $disk below.
* voldisk rm $disk
* Now replace the hard drive.
* hwmgr -show scsi -> take a note of your current set of disks.
* hwmgr -scan scsi
* hwmgr -show scsi -> the replaced disk should show up as a new disk at the bottom of the list. This'll be used as $newdisk below.
* dsfmgr -e $newdisk $disk
* disklabel -rw $disk
* voldisk list -> $disk should be labeled as "unknown" again.
* voldiskadm -> option 5 -> $vmdisk -> $disk -> y -> y -> your VM disk should now be replaced.
* volrecover -g $diskgroup -sb

The remirroring process will now start for all broken mirrors. Unfortunately there is no way of tracking the actual process. You can check whether the mirroring's still running with "volprint -ht -g $diskgroup | grep RECOV", but that's about it.

Posted in: lsm , sysadmin , tru64 , work

As promised: adding a new LUN to Tru64 (Kilala.nl (Cailin Coilleach)) by Cailin Coilleach

As I promised a few days ago I'd also give you guys the quick description of how to add a new LUN to a Tru64 box. Instead of what I told you earlier, I thought I'd put it in a separate blog post instead. No need to edit the original one, since it's right below this one.

Adding a new LUN to a Tru64 box with TruCluster
1. Assign new LUn in the SAN fabric.
No something I usually do.

2. Let the system search for new hardware.
hwmgr scan scsi

3. Label the "disk".
disklabel -rw $DISK

4. Add the disk to a file domain (volume group).
mkfdmn $DISK $DOMAIN

5. Create a file set (logical volume).
mkfset $DOMAIN $FILESET

6. Create a file system.
Not required on Tru64. Done by the mkfset command.

7. Test mount.
Mount.

8. Add to fstab.
vi /etc/fstab

Also, if you want to make the new file system fail over with your clustered application, add the appropriate cfsmgr command to the stop/start script in /var/cluster/caa/bin.

Posted in: learning new stuff , lvm , san , tru64 , work

Crash course in new OSes (Kilala.nl (Cailin Coilleach)) by Cailin Coilleach

The past two weeks I've been learning new stuff at a very rapid pace, because my client uses only a few Solaris boxen and has no Linux whatsoever. So now I need to give myself a crash course in both AIX and Tru64 to do stuff that I used to do in a snap.

For example, there's adding a new SAN device to a box, so it can use it for a new file system. Luckily most of the steps that you need to take are the same on each platform. It's just that you need to use different commands and terms and that you can skip certain steps. The lists below show the instructions for creating a simple volume (no mirroring, striping, RAID tricks, whatever) on all three platforms.

Adding a new LUN to a Solaris box with SDS

1. Assign new LUN in the SAN fabric.
Not something I usually do.

2. Let the system search for new hardware.
devfsadm -C disks

3. Label the "disk".
format

4. Add the disk to the volume manager.
metainit -f $META 1 1 $DISK

5. Create a logical volume.
metainit $META -p $SOFTPART $SIZE

6. Create a filesystem
newfs /dev/md/$META

7. Test mount.
mount $MOUNT

8 Add to fstab.
vi /etc/vfstab


Adding a new LUN to an AIX box with LVM

1. Assign new LUN in the SAN fabric.
Not something I usually do.

2. Let the system search for new hardware.
cfgmgr

3. Label the "disk".
Not required on AIX.

4. Add the disk to a volume group.
mkvg -y $VOLGRP -s 64 -S $DISK

5. Create a logical volume.
mklv -y $VOLNAME -t jfs2 -c1 $VOLGRP $SIZE

6. Create a filesystem
crfs -v jfs2 -d '$VOLNAME' -m '$MOUNT' -p 'rw' -a agblksize='4096' -a logname='INLINE'

7. Test mount
mount $MOUNT

8 Add to fstab.
vi /etc/filesystems


Adding a new LUN to a Tru64 box running TruCluster

I'll edit this post to add these instructions tomorrow, or on Friday. I still need to try them out on a live box ;)


Anywho. It's all pretty damn interesting and it's a blast having to almost instantly know stuff that's completely new to me. An absolute challenge! It's also given me a bunch of eye openers!

For example I've always thought it natural that, in order to make a file system switch between nodes in your cluster, you'd have to jump through a bunch of hoops to make it happen. Well, not so with TruCluster! Here, you add the LUN, go through the hoops described above and that's it! The OS automagically takes care of the rest. That took my brain a few minutes to process ^_^

Posted in: aix , solaris , tru64 , volume management , work