Blackouts and Battery Backups

With the recent flooding and blackouts here in Toronto, Ontario, Canada, I’ve decided to replace my aged UPS (my computer’s battery backup) with a fresh one from Costco and while there, I was pleasantly surprised to find they had one from what most people consider to be the biggest name in the business, APC, and it turned out to be a good model for the price they asked. They had an APC Back-ups x1300 for under $200.

After bringing it home to protect my AD virtuals on my ESXi tower and my PBX and NAS I found it only allowed my to use and only would talk to, their Personal Edition of their software PowerChute which meant it would only be monitored by a single installation of an OS, in this case Windows 7.

After a little playing with it I found how to make the USB attached detection transfer through ESXi to the virtual Windows 7 and so began my battle to make it shut down everything.

I started with the basics. I wrote a simple batch file to say Hello World and triggered the battery by unplugging the UPS from power for over minute and I found a specific event in the Windows Event Logs ‘174’ was the Event ID and it seemed to happen whenever the UPS switched over to the battery.

So I right-clicked that and was given an option the trigger a Windows task based on this event happening again.

A note to anyone playing with Windows tasks in Vista /2008 or higher. In order to use the option to have the event trigger even if the needed user is not logged in, you will need to play with your Windows Group Policies to give that user the rights to ‘log on as a batch’, otherwise it will just fail with a permissions error.

After spending about on hour playing with the task’s settings to make it run regardless to the user being logged in, I tested it with another pull of the power cable and found it does work to start a program, or batch file in my case, but that nothing shows up on my screen in any case so I had it log output to a file to confirm it was active.

Windows task settings did give me an option to have a message popup on the screen and send out emails first but I could not get either of those to work in a measurable way.

I worked my way through a batch file which would send the ‘shutdown’ command to other Windows systems and for this to work I had to switch the user I was running the task as to a user with that kind of permission on all the Windows VMs. Thankfully they are all on the same Windows domain so a domain administrator was easy enough to use to run the task.

I then put in a 60 second sleep count and used plink.exe from the putty suite to trigger a shell script on my PBX server. This is useful because the Windows 7 virtual running this task during a blackout also needs to shutdown while more happens, so the 60 second sleep timer allows the Windows VMs to finish powering down and then a script to continue everything starts up on my PBX which is on CentOS and on its own Acer Revo hardware. So I finished this batch file up with a simple shutdown now command to Windows and it was ready to test.

The result was the UPS kept things running fine with a 40 minute counter to start and within 2 minutes all the VMs on my ESXi tower had been shutdown including the domain controllers and the WSUS.

So now I started to work from the Linux side of things and I found that from Linux you can just type the command SSH and it works but it always asks you for a password from command line and this made it hard to script so I googled and enable the RSA key authentication between specific hosts internally and no more asking for a password from a specific host to other Linux systems I have.

Now came the fun of trying to script an Ubuntu server to shutdown remotely. I googled how to use sudo over SHH but scripted, it would not work. Then I hit upon the idea of modifying the sudo access on the Ubuntu VM so that my specific user does not need a password to run ‘sudo shutdown’ and every article I found talked about editing the suckers file but not a single one worked. I ended up remembering that I did get the VMware Tools to work on the Ubuntu VM so from SSH to the ESXi host, I was able to script it to send a nice shutdown command to the Ubuntu, then my SmoothWall Firewall/Gateway, then push ESX into maintenance mode (required for the next step), then shutdown ESXi completely.

Then remembering I had already installed SSH on my NAs ( a D-Link DNS-320) I just scripted that shutdown command higher up so it starts sooner.

Following all of this the only device that is left standing is the PBX server itself, so I finished off the script with a simple ‘halt’ command and it was done.

Upon further testing I also noticed that when power gets restored, some of the device boot up automatically and this was wonderful news.

My next project is to automate the startup after a blackout, but that’s a story for another blog post.