. FTE’s responsible for providing technical support (Frontline and backline included) for VCS,VxVM, VOM, Data Insight, Storage Foundation and VERITAS Products etc. Manage and integrate a wide variety of technical services within one center to provide support to a designated customer population within defined geographical regions. Veritas has re-branded it’s “Storage Foundation and High Availability” family products and now, from version 7 all the of the Veritas products under this category comes with a brand name of “Infoscale“.In this tutorial, we are going to teach how to create a Veritas Volume Manager (VxVM) volume and then how to create a VxFS filesystem and mount it in Red Hat Enterprise Linux 7 (RHEL 7). VxVM, on Solaris 10 for SPARC, has been set up to mirror each disk, ie Array 1 Disk 1 is mirrored to Array 2 Disk 1 etc etc. We had a problem with the power to the controller of array 2 and the disks in that array ended up in a disabled and removed state. The power has now been restored and the disks are now visible to the operating system again. Replacing a failed disk under VxVM Let us now see different volume layouts that can be created in veritas volume manager. Before diving deep into the RAID levels first create a Disk group with 4 disks. To do this we have to initialize 4 disks to take them under VxVM control. Vxassist(1m) is a vxvm command that acts as a front-end to other vxvm commands, much as newfs(1m) acts as a simpler-to-use front-end to mkfs(1m). Using vxassist with it's -v flag will display the actual vxvm commands used during volume construction. Vxassist also front-ends vxfs commands and normal UNIX utilities during volume construction.
Vxvm Tutorial
Veritas Volume Manager
VM daemons
| vxconfigd | Maintains system configuration in the kernel & disk (private region). If the daemon is stopped it does not disable any configuration state loaded into the kernel, it only affects the ability to make configuration changes until vxconfigd is restarted. It can be in three states: |
| vxrelocd | Monitors for failure events and relocates failed subdisks |
| vxconfigbackupd | Used to backup configuration chnages, the files created can be used with vxmake to restored lost groups. |
| vxnotify | Display veritas volume manager events used with the vxconfigd daemon |
| vxcached | used to administer a cache object that is assoicated with volumes that have one or more space-optimized snapshots. When the usage of a cached volume reaches the high watermark vxcached autom,atically grows the cache volume if required and configured |
| vxattachd | Monitors VxVM for disks being attached and reattaches a detached site if the disks belongs to that sire become accessible, the daemon monitors vxnotify command and waits for a failed disk, when the disk is attached vxattachd attempts to online the disk, if successful it then starts recovery using vxrecover |
| vxdbd | Handles comunication to and from the storage foundation product, it uses port 3233 |
| Kernel States | The kernel can be in three states: Enabled - both private and public regions are accessible |
VM utilities
| VxVM debug | vxconfigd -k -m enable -x <debug option> -x log log to /var/vxvm/vxconfigd.log |
| vxiod | The vxiod utility starts, stops, or reports on VERITAS Volume Manager (VxVM) I/O daemons. An I/O daemon provides a process context for performing I/O in VxVM.Manage extended disk i/o & handles dirty regions, logging vxiod set <number> = set number of runnning viod daemon Note: when run on its own it displays # of vxiod daemons that are running. |
| vxdctl [option] | The vxdctl utility manages aspects of the state of the volume configuration daemon vxconfigd and also manages aspects of configuration for bootstrapping the rootdg disk group. mode = what mode the vxconfigd is running in Note: when in disabled or stopped mode no VX commands will be able to run |
| vxinstall | Install volume manger (use /etc/vx/disk to exclude any disks or controllers) |
| vxdbdctl | used to stop/start the vxdbd daemon, you can also use status to obtain the status of the daemon |
Disk Regions
Disk Regions | ||
| Private | This is were veritas holds the meta data regarding the disk. A copy of the configuration database is copied to each private region within the disk group. Veritas will try and keep 5 copies of the configuration database. See below for where the slice is kept Once the private region is created it cannot be changed, thus if it fills up then you have to reinitize the disk with a new private region length, see disks -> initialize disk | |
| Public | This is the area that will store the users data. See below for where the slice is kept | |
| Sliced Disk layout | private and public region slices are on separate partitions, this type of disk is not suitable for moving between different O/S's but are suitable for boot partitions Can be converted to CDS | format=sliced |
| CDS (Cross-platform Data Sharing) | private and public regions are one slice , this type is suitable for moving between different O/S's but not suitable for boot partitions. | format=cdsdisk |
| Simple | Private and public are the same partition but continuous Can be converted to CDS | format=simple |
| None | No partitioning | n/a |
VxVM Configuration Database
| Public/Private partitions | vxdisk list <disk> |egrep -i '^public|^private' |
| DB size | # the size of the configuration db vxdg list <group> | grep permlen |
| DB location | vxdisk list <disk> | grep -i config - db location |
File Locations
| vxinstall has not be run | /etc/vx/reconfig.d/state.d/install-db |
| Host ID's | /etc/vx/volboot |
| backup config files (vxconfigbackupd) | /etc/vx/cbr/bk Note: you must have at least run vxconfigbackup once, otherwise the directory does not exist or you have specified another location. |
| delete or deported disk group config files | /etc/vx/dgcfg/deport |
| All commands logs | /var/adm/vx/cmdlog /var/adm/vx/veacmdlog |
| Licenses | /etc/vx/licenses/lic |
| Imported disk groups info | /var/vxvm/tempdb |
| vxconfigd log file | /var/adm/vx/vxconfigd.log |
Backup & Restore
| Backup | /usr/lib/vxvm/bin/vxconfigbackup -l /var/vxvm/backups -l = location where to store backup |
| Restore (precommit/commit) | vxconfigrestore -p <group> # either one of the below after the precommit Note: |
Disks
| Initialize disk | ## increase the private region size default 1024 ## change the default of a disk Note: format can be either sliced, simple, cdsdisk or none (see above - Disk Regions) |
| Uninitialize disk | vxdiskunsetup -C c2t0d0 |
| Disk Information | vxdisk -g <group> list <disk> vxdisk -s list |
| Resize a LUN | vxdisk -g <group> resize <disk> length=8G |
| Add a disk slice to volboot | vxdctl add disk <device> type=simple |
| Add a disk slice | vxdisk –f <device> type=simple |
| Add a disk | vxdiskadd c1t0d0 or c1 (all disk on controller) vxdisksetup -i <device> |
| Remove a disk totally from VM | vxdisk rm <device> |
| Remove a disk from a volume | vxdg -g <group> rmdisk <diskname> |
| Remove a disk slice from VM | vxdctl rm disk <device> |
| Clear any host ID flags | vxdisk clearimport <disk name> |
| Renaming a disk | vxedit -g <disk> rename <old disk name> <new disk name> |
| Move disk to different disk group | vxdg move <source dg> <target dg> <disk> Note: you must always have one disk in a disk group, you cannot move a disk that is used by a volume |
| Offline a disk | vxdisk offline <dev name> Note: disk must not be in a disk group |
| Online a disk | vxdisk online <dev name> Note: disk must have a private region otherwise you need to initialise the disk |
| Hot spare | vxedit –g <group> set spare=on <disk> |
| NoHotUse | vxedit –g <group> set nohotuse=on <disk> |
| Turn off failing flag | vxedit -g <group> set failing=off <disk> |
| Encapsulate a disk | vxdisk define c0t0d0s0 type=nopriv |
| Reattach disk (SAN) | vxreattach [-br | -c <device>] -b = Background process |
| Discover new disks | vxdisk scandisks [new | fabric ] |
| Disk Comment | vxedit -g <group> set comment='.......' <disk> |
| Private region problem | ## Here i am fixing a 'online altused' issue but search the web for different issues as there are a number of ways to the same thing, just remember the vxprivutil command ## obtain the disks private region tag 15 ## run the below script ## if now errors reported above and in the file, then dd back to disk ## repeat for any other bad disks |
Disk Group
| Create a disk group | vxdg init group <disk>=<device> vxdg init <group> <disk>=<device> cds=off - initialize a non-cds disk group |
| Remove a group | vxdg destroy <group> |
| Add a disk to a group | vxdg –g <group> adddisk <disk>=<device> |
| Remove a disk from a group | vxdg –g <group> rmdisk <disk> |
| Replace failed disk | vxdg -k -g <group> adddisk <disk>=<device> -k = forces vxvm to take media name of the failed disk & assign it to the new disk |
| Import a group | vxdg import <group> |
| Import group (clear any flags) | vxdg import -C <group> -C - clears any exist host flags |
| Deport a group | vxdg deport <group> vxdg -n <new-group-name> deport <old_group_name> |
| List no hot use on disk | vxdg –g <group> nohotuse |
| List spare space on disk | vxdg –g <group> spare |
| Display free space | vxdg –g <group> free |
| Backup disk group (vxvm 4.0) | vxconfigbackup |
| Restore disk group (vxvm 4.0) | vxconfigrestore [-p|-d|-c] |
| Diskgroup Version | vxdg list <group> | grep -i version |
| Upgrade disk version | vxdg upgrade <group> - upgrade to current version vxdg -T 50 upgrade <group> - upgrade to version 50 vxdg -T 50 init <group> <disk>=<device> - creater new group @ version 50 |
| Boot/Default DG | vxdg bootdg vxdg defaultdg vxdctl defaultdg <group> - set defaultdg |
Volume
| Adding mirror to root | /etc/vx/bin/vxrootmir <alternate> create rootvol, swap vol, |
| Create a simple volume | vxassist make <volume> <size> <disk> vxassist make <volume> <size> <disk> layout=[stripe-mirror|concat-mirror|mirror-concat|mirror-stripe] stripe-mirror = layered volume |
| Mirror a simple volume | vxassist mirror <volume> <disk> vxassist -g <group> remove mirror <vol> !disk01 - remove the disk01 mirror |
| Create a stripped volume | vxassist make <volume> <size> layout=stripe vxassist -g <group> -o ordered make <vol> <size> layout=stripe ncol=3 <disk1> <disk2> <disk3> |
| Create mirrored volume with log | vxassist make <volume> <size> layout=mirror, log nmirror=# nlog=# Note: for information about logging see logging section below |
| Create a raid volume | vxassist make <volume> <size> layout=raid5 |
| Create a raw volume | vxassist -U <usage_type> make <volume> <size> alloc='<disk>' types: |
| Remove a volume | vxedit –rf rm <volume> Note: you must disable the volume first |
| Initializing a volume | vxvol init state <volume> [plex]state=clean,enable,active |
| Online Relayout | vxassist -g <group> relayout <vol> layout=stripe ncol=2 # Display the relayout operation |
| Remove a volume off a particular disk | ## either make sure there is a another disk in the group, you can even specify it vxassist -g <group> move <vol> !<disk> [<disk>] |
| Rename a volume | vxedit -g <group> rename <old_vol_name> <new_vol_name> |
| Starting a volume | vxvol start <volume> |
| Start a disabled volume | vxrecover –sb <volume> -s = start volume after recovery |
| Disable a volume | vxvol -g <group> stop <volume> |
| Evacuate a volume | vxevac -g <group> <from-disk> <to-disk> |
| Maintenance mode | vxvol maint <volume> |
| Not clean | vxmend mirror clean <plex> |
| No kernel state | vxplex att <vol_name> <plex> |
| Extending a volume size | vxresize <volume> <new length> vxresize -g <group> <volume> +100m - increase the volume by 100Mb |
| Shrinking a volume size | vxresize <volume> <new length> vxresize -g <group> <volume> -100m - decrease the volume by 100Mb |
| add a DRL log to a volume | vxassist addlog <volume> |
| remove a DRL log from a volume | vxassist remove log <volume> |
| Extending log size | vxvol set loglen = 2m <volume/log> |
| Detering volume size | vxassit -g <group> maxsize layout=mirror - the maximum size you can create a mirror vxassist -g <group> maxgrow <volume> - the maximum size the volume can grow too |
| Recover a volume | Vxmend fix clean <plex> |
| Change volumes permissions | vxedit -g <group> set owner=<user> group=<group> mode=<perms> <vol> # to display the permissions ls -l /dev/vx/rdsk/<group> |
Plexs| Creating a plex | vxmake plex <plex> sd =<sub disk name> |
| Remove a plex | vxplex –o rm dis <plex> vxplex -g <group> dis <plex> vxedit -g <group> -rf rm <plex> |
| Moving a plex | vxplex mv <original plex> <new plex> |
| Copying a plex | vxplex cp <volume> <new plex> |
| Attaching a plex | vxplex att <volume> <plex> |
| Detaching a plex | vxplex det <plex> |
| Offlining a plex | vxmend off vol01-02 |
Sub-disks
| Creating sub-disk | vxmake sd <sub-disk> <disk>, offset, len |
| Removing sub-disk | vxedit rm <sub-disk> |
| Moving sub-disk | vxsd mv <old sub-disk> <new sub-disk> |
| Associating with a plex | vxmake plex <plex> sd=<sub-disk>, … i.e vxmake plex home-1 sd= disk02-01, disk02-00, disk02-02 |
| Dissociating | vxsd dis <sub-disk> |
| Splitting | vxsd –s<size> split sd<new sub><newsub2> |
| Joining | vxsd join <sub-disk1><subdisk2><new subdisk> |
| relocating a sub disk | ## either make sure there is a another disk in the group, you can even specify it vxassist -g <group> move <vol> !<disk> [<disk>] |
| relocating a whole disk sub disks | vxprint -g rootdg -se 'sd_orig_dmname='disk02' |
Volume Manager Information
Disks | |
| Display all the physical disks | vxdisk list |
| Display detailed disk info | vxdisk list <disk> |
| Check for disk failures | vxstat –g <group>–ff –d |
| Display group properties | vxdg list |
| Display detailed group info | vxdg list <group> |
| vxinfo -p -g <group> | |
Volume | |
| Display volume info | vxprint –Aht <vol> |
| Display volume properties | vxprint –vl |
| Display unstartable volume | vxinfo -g <group> <vol> |
| Check for volume failures | vxstat –g <group>–ff -v |
| Display plex properties | vxprint –vp |
| Check for plex failures | vxstat –g <group>–ff -p |
Sub-Disks | |
| Display sub-disk properties | vxprint –st |
| Display tasks | vxtask list States: |
Statistics and Tracing | |
| Iostats | vxstat -g <group> -r –d <disks> - reset all stats on disk vxstat -g <group> -d - display stats vxstat -g <group> -i 1 -d <vol> - display stats every 1 sec intervals for volume vxstat -g <group> -i 10 -c 5 -d - display 5 sets @ 10 secs intervals |
| Tracing | vxtrace -d <filename> -o dev,disk <vol> vxtrace -f <filename> -o dev,disk <vol> | more |
Licensing
| Add | vxlicinst - versions greater than 3.5 vxlicense -c - versions below 3.5 |
| View | vxlicrep - versions greater than 3.5 vxlicense -p - versions below 3.5 |
| Paths | /etc/vx/licenses/lic - versions greater than 3.5 |
| Reload new license | vxdctl license init |
VEA
| Start/Stop | /etc/init.d/isisd [start|stop|restart] |
| Status | vxsvcctrl status -m = status |
| Daemons | /opt/VRTSob/bin/vxsvc /opt/VRTSob/bin/vxsvcctrl |
| Start VEA GUI | /opt/VRTSob/bin/vea You must have X-windows running |
Logging
Logging help in recovery and can speed it up dramatically, the main form of logging in veritas is the DRL (dirty region log) which performs the following
- log keeps track of changed regions
- if system fails only the changed regions of the volume are recovered
| Add | vxassist -g <group> addlog <vol> logtype=drl vxassist -g <group> addlog <vol> - used for raid logs (no type) |
| Remove | vxassist -g <group> remove log <vol> [nlog=n] <vol> |
Volume Read Policy
Policies can be used if you have slower disks within a volume and you wish to use the faster disks.
| Round Robin | vxvol -g <group> rdpol round <vol> |
| Preffered Plex | vxvol -g <group> rdpol prefer <vol> <plex> |
| Selected Plex | vxvol -g <group> rdpol select <vol> |
Storage Expert (vxse)
Veritas have created some scripots that can check the integrity of the vxvm setup i.e mirrored volumes, spares, etc. The scripts are based on rules and there are a number of differents rules veritas has set, look in the rules directory to see all of them.
Vxdiskadm
| Display Description | vxse_raid5log1 info |
| Check rules | vxse_raid5log1 -g <group> check |
| List spare rules | vxse_spares list |
| Run spare run | vxse_spares run Note: you need to run '/etc/init.d/isisd start' to start the necessary daemons first |
| Rules | /opt/VRTS/vxse/vxvm |
| Default Parameters | /etc/default/vxse |
Vvm Volume
VxDMP
see VxDMP for more information
Vxvm Cheat Sheet
Veritas links
Veritas Volume Manager
| Recovery features | http://prefetch.net/articles/veritasrecoveryfeatures.html |