Wrong Place, Wrong Time

Earlier I mentioned that I had changed jobs and this is one of those cases where it’s become obvious that the move was just the wrong place at the wrong time.  When I was working I absolutely loved the job, and how could I not when it’s all VMware all the time.  I somehow managed to score almost exclusive SRM jobs which made my life all the better.  Unfortunately there’s a lot more in a career change than just the work being performed.  Enough said, I hope.

There was an opening at a local company for a position in which I had written the job description that I decided to take after some discussion and an interesting offer.  Yep, I’m going back to the place I left taking on the same role as a virtualization and storage engineer.

I sort of feel like I’ve come home again, it’s a nice feeling.  I’m going to miss the opportunities presented in the role of a consultant and I’m sure I’ll be back at some other time in some other place.

First Impressions of Life as a Consultant

I can say for sure that it’s not quite what I expected.  I’m not entirely sure what I did expect but I knew that I’d love the technology and the challenge ahead.  I was pretty sure my level of knowledge was not quite sufficient for the job but looked forward to the pressure to learn quickly.  The travel isn’t for me and I knew this going into the job.  I have accepted the travel and cannot complain.  I expected to be a billing machine, this is no surprise here and, again, I cannot complain.

What I didn’t quite realize, call it just being naive, was that some IT professionals don’t necessarily want to have a consultant in their shop.  My experiences and perception is the exact opposite, though I always was the one to make the decision as to whether to purchase PSO for an implementation project or not.  It’s no surprise that many of my peers don’t necessarily get to make that decision, I still find the response to my presence as a consultant surprising.  This is where one needs to have people skills and be able to form a bond with others quickly, I’ve found that taking them out for happy hour helps!

I do have some concerns with the methods in which I was dropped into my first jobs though will not go into it deeply for obvious, professional reasons.  I do have to say that it’s a damned good thing that all jobs to date have been on the one VMware technology that I’m extremely comfortable with, Site Recovery Manager.

The exposure to different environments is amazing, even better than I thought it would be.  The inability to get work done in an infrastructure that’s not mine kills me.  I can’t stand waiting on something like having a LUN provisioned, zoning all paths to storage instead of just one, making decisions on virtual networking, creating a portgroup, installing a patch, etc.  I haven’t yet worked in an environment where I could not either do or facilitate work.  My experiences are all SMB and I know that this is extremely different in some large shops, it not a surprise but still frustrating.

Mostly, I miss having that infrastructure that I can claim ownership of.  This I did not anticipate at all.  There is no vSphere/storage environment that I am responsible for, that I can assess and dream up changes to make it better, and nothing that someone else is holding me accountable for.  Might seem odd to some but I love having that which I can own and am expected to maintain for highest performance and reliability.

Sure, implementation projects are cool.  I still miss some of the day to day work that I’ve had in the past.  I am very sure that I only miss this because of my smaller environments and due to the fact that I could own my environment.  I had HBA to physical disk including any virtualization layer in between and more.  Now I have a project to plan and execute on.  Don’t misunderstand, I love the opportunity to architect a project just as much as the next IT professional, or I suspect we all like this part of it.  I just miss that which was mine and I’m not so sure what to do about that.

This does not mean that I don’t like my job, quite the opposite.

Mike Laverick’s Administering VMware Site Recovery Manager 4.0

Ridiculously Overkill Windows Server 2008 Template Preparation

Following are the steps I took to attempt to optimize a Windows Server 2008 Template, recorded largely for my own benefit.  Optionally, replace SCSI device with PVSCSI device to be outlined elsewhere on this blog.

Please feel free to comment for logic errors, corrections or additions.  Keep in mind that this is Server 2008 specific.  Many of the actions apply to Server 2003 as well, locations might be different and file system block alignment would need to be addressed.

1. Edit Settings -> Options -> Boot Options -> Power-on Boot Delay = 5000
2. Select Force BIOS Setup
   1. Disable Serial and Parallel ports, BIOS -> Advanced I/O -> Disable
   2. Set Boot options, BIOS -> Boot -> CDROM 1^st, Hard Disk 2^nd
3. Reboot
4. Install VMware tools, Custom option
   1. Disable “Shared Folder” driver
   2. Disable “Audio Driver”
   3. Disable pvSCSi driver is not needed
   4. Disable SVGA drivers installed with VMware Tools on Windows 7 and Windows 2008 R2. See this KB article for Windows 2008 R2 or Windows 7
5. Control Panel settings
   1. Enable hardware acceleration, Personalize -> Display Settings -> Advanced Settings -> Troubleshoot -> Change Settings -> Full Hardware Acceleration
   2. Disable Screen Saver, Personalize -> Screen Saver -> None
   3. Sound, Do not start Audio Service -> Sound
   4. No Sounds
   5. Disable “Play Windows Startup Sound”
   6. Display Performance, System -> Advanced System Settings -> Performance Settings -> Adjust for best performance
   7. Power Settings
      1. Power Options -> High Performance
      2. Power Options -> Change when the computer sleeps -> Turn off display = Never
   8. Pagefile, System -> Advanced -> Performance -> Set no pagefile
6. Disable System Screen Saver (screen saver that starts when no user is logged into the system) -> Regedit -> “HKEY_USERS\.DEFAULT\Control Panel\Desktop” -> Delete SCRNSAVE.EXE
7. Insure Audio Service is stopped in Service MMC, set to Manual
8. For ultimate performance on C:\, when absolutely necessarily, move pagefile to separate drive.  PVSCSI device maybe
9. Remove IPv6 Support (optional)
10. Start Menu -> Remove Quick Launch
11. Drive Indexing, My Computer -> C:\ Properties -> Unselect “Index this drive…” -> Apply recursively -> Ignore all permissions errors
12. Run as Administrator ‘cmd.exe’ -> ‘powercfg.exe –h OFF’ to disable hibernation and delete hiberfile.sys from C:\
13. Reboot
14. Delete pagefile.sys on C:\
15. Defrag C:\
16. Reset pagefile
17. Clear C:\Users\Administrator\AppData\Local\Temp folder
18. Disable Windows Firewall (optional)
19. Enable Remote Desktop
20. Activate EA licensed Windows

edit: Of course, VMware Tools and hardware version upgrade to 7 is assumed!

Add VLAN tagging to COS and VMkernel portgroup

I had need to tag the service console and vmkernel portgroups well after the fact of building a bunch of ESX server.  It’s actually pretty easy to do.  Command is: esxcfg-vswitch -p <Portgroup Name> -v <VLAN ID> <vSwitch Name>

`esxcfg-vswitch -p Service\ Console -v 999 vSwitch0`

From there the easy way of change the VMkernel port is via the GUI.

Career Changes

This isn’t nearly as glamorous as the flurry of announcements being made about the joining of Chad’s vArmy but is a big one for myself.  I’ve just made the move on to a new employer, Data Strategy, as a Consultant focusing on VMware technologies.  This is a move I’ve been looking at making though it’s occurring much sooner than anticipated.  I see this, consulting for VMware that is, as the next step in advancing my career and enhancing my breadth/depth of knowledge with VMware products.  I will be updating my LinkedIn page in the very near future, if you haven’t connected to me yet please do feel free.

Resize a non-LVM ext3 Partition After Increasing Size of Virtual Drive

1. Resize the virtual disk
2. Reboot
   1. yes it can be done without a reboot but guys far smarter than myself strongly discourage it!
3. Unmount the drive
   1. umount /dev/sxxp
4. Delete and recreate the partition table
   1. fdisk /dev/sxx
   2. d to delete (do not write)
   3. n to create new
   4. p for primary
   5. 1 for 1st partition
   6. accept remaining default for first and last blocks
   7. w to write
5. Reread partition table
   1. hdparm -z /dev/sxx
6. Check the filesytem
   1. e2fsck -f /dev/sxxp
7. Resize the filesystem
   1. resize2fs /dev/sxxp
8. Mount the filesystem, assumes mount definition is in /etc/fstab
   1. mount /dev/sxxp

vSphere Upgrade, Part III

Quick notes on the SRM 4.0 install, which was also pretty trivial save for a few key points.  I needed to take advantage of the Shared Recovery Site feature, well documented on VMware’s Knowledge Base article 1014640.

The real short story on the install is to execute the installable via command line with a /V”CUSTOM_SETUP=1″ flag.  Unfortunately, Windows 2008 complicates this slightly, I executed cmd with “Run as Administrator” privileges to overcome.  Aside from creating a shared custom site identifier I just followed the prompts.  I did find notes regarding database connectivity, after troubleshooting service failures, in the Administrators Guide:

If you are using Windows authentication, the SRM server and database server must run on the same host.

If the SRM server and database server run on different hosts, you must use mixed mode authentication.

Database logging was showing that the local service account was attempting to initialize the database connection despite the fact that the SRM service was running as a domain user with a database login.  A switch to SQL Server Authentication resolved all issues the quick and dirty way.  After the installation I was able to immediately install the vCenter Client SRM plugin and pair my sites.  It was that easy.

After upgrading the ESX hosts to 4.0 I quickly found that snapshot LUN were no longer visible on the host servers.  This makes perfect sense though I had completely forgotten about it and how to reconfigure this.  All thanks to a quick tweet from @vseanclark I found that the LUN has to be resignatured manually from the recovery servers.  The process goes something like this, in RecoverPoint terminology:

1. Enable image access
2. Rescan with recovery ESX hosts
3. “Add Storage” on ESX host
4. Select “Resignature LUN”, or something to that effect

It appears as though I only had to perform this on one host in the cluster and I believe that this actually makes perfect sense.

vSphere Upgrade, Part II

First, there’s been a slight change in my design, I’ve moved vCenter services to a virtual machine. I found, during re-reads of the documentation, that Linked Mode is not supported in clustered vCenter Servers and being the lazy admin that I am I wanted that convenience. For now I just renamed my Update Manager server and installed vCenter on it. I might split off the UM role, undecided at this time and this is something I can totally deal with at any time. My big push is to just get this done.

I’m going to use this to outline the specific actions taken through my process of upgrading vCenter in my environment. Later, I will follow up with SRM, ESX and maybe Lab/Lifecycle/Stage Manager servers.

As noted in my previous post my database is an active/passive Failover Cluster on physical IBM HS21 blades. This hardware is not new, it’s actually old ESX servers that I decommissioned from my DEV environment a couple of years ago. I believe they are a dual Intel 5150 dual cores, non-XM model with a grand total of 8 GB RAM and a pair of 74 GB hard disk drives. I have these blades in the same chassis as 10 of my ESX servers in production and am sharing a chassis with DR servers for my second vCenter database cluster.

My DR sites shares floor space with TEST and DEV resources and, in the near future, will be hosting other PROD resources. DR resources share an EMC CX4-480 with TEST. This datacenter is built up in my office, we are going to be deploying new A.D. servers, relocating file servers and NAS hosted CIFS initially.

The SQL Server cluster built was fairly straight forward, so much so that even I as a non-Microsoft guy could do it. My build was on Server 2008 Enterprise with SQL Server 2005 Standard Edition, the following documentation is applicable this environment only.  I eventually got the installation just the way I wanted it on the 3rd or 4th build. What I finally found, which countered a lot of the documentation out there, as the most effective steps are: Read the rest of this entry »

vCenter U1 upgrade and SRM failure

In order to provide support for my Windows 7 using administrators I made the hasty decision to do a quick upgrade on vSphere Server to U1. After the upgrade I found that SRM services would no longer start on my SRM servers.  After spending way too long on troubleshooting what appeared to be invalid credentials I finally came across an article that reminded me of the new “repair” feature in Site Recovery Manager.  About that point it really hit me that credentials were fine, it must have been the certificates that were not.  Of course I didn’t back up the SSL files!  While attempting to run the repair I discovered something else; on Windows Server 2008, even when logged in as a local administrator, that using the Programs and Features snapin is not sufficient for running the repair.  I found the most efficient method appears to be executing the installable wtih “run as administrator” feature, selecting “repair” and following the prompts.

This immediately resolved the conflict.