On Tuesday morning, at about 3.15AM, I had a phone call from the out of hours service that there was an issue with one of our systems. After fumbling around for a half hour I worked out that I was not going to be able to fix the issue remotely and would have to go on-site to fix.
Unfortunately when I got onto site, it was discovered that one of the HP MSA arrays that is connected into part of our vSphere 4 environment had lost all power. Because of the half-arsed way that the consultancy company set it up (a process I was not involved with!) it took out the other enclosure with it.
Anyway, once power had been restored to the system, the vDisks were showing as offline and no amount of rebooting (at HP’s request) would bring them back online. Eventually the case was escalated to their level 2 team and I was asked to do the following:
- Disconnect (or power off) all hosts connected to the MSAs
- Telnet to the MSA that had lost power.
- Enter the command: trust enable
- Enter the command: trust vdisk vdiskname
- Wait for a minimum of 3 minutes.
- Enter the command: show vdisks
The disk with the name vdiskname should then appear as FTOL, and if not the commands above should be run a second time. This then needs to be run for all vDisks that have been set up on the MSA. I was told by HP to only run these commands when advised by HP, but I thought they may prove useful to someone somewhere.
And yes, I’m very pissed off with the consultants that set it up!!!