A few weeks ago, I wrote a post called “When Things Go Horribly Wrong. Again.” In that post, I talked about our lighting console crashing (turns out we had a bad SSD), and how we recovered from it. That crash brought to light some major gaps in our backup process; ones I now hope we’ve closed.
In the weeks leading up to the backup, I thought everything was fine. We had an external backup drive attached and backup software that was supposed to be backing up every Saturday night. Turns out “supposed to” is the operative phrase. Apparently, it wasn’t backing up, and hadn’t for quite some time. I don’t know why.
Normally, I’m a belt and suspenders kind of guy. I don’t consider any files safe unless they’re on at least two preferably three) drives or media, with one in the cloud. I like the 3-2-1 backup strategy (the link takes you to a PDF report of that strategy) that was developed in conjunction with the National Archives.
All files should exist in three different places, on two different media with at least one off-site. I do that with our audio console show files, but we never set it up for lighting. Now we have. Here’s what we did.
Backup, Backup, Backup
It turns out that in the new version of Hog PC (v. 4), they included a handy-dandy backup button. It’s a one-stop-shop for making a backup of your show file. Somehow, it makes the backup files very small, but it’s still the whole show file. How can we not take advantage of that? Turns out, we weren’t. Now we do.
The standard shut down procedure on Saturday night is to first hit “Backup” to make a backup show file. That way, if things go south on Sunday, we can get right back to where we were. But that only protects us from a show file crash; if the hard drive goes again, we’re hosed. So, we back up the backup.
Dropbox For The Win!
Dropbox is perhaps the easiest way to back things up. Now, we could go into the Hog’s backup folder and manually copy the latest backup over to the Dropbox folder and let it do it’s thing. But that takes time and is a pain. So I automate.
Using a free utility from Microsoft called Sync Toy, I built a little sync script that will, when fired, sync the backups folder with the Lighting Backups folder in Dropbox. And again, we could launch Sync Toy manually, run the sync, wait for Dropbox to upload, then shut down. Again, a pain. And boring. So, we automate once more.
I stole a little batch file that Isaiah Franco made for me back when he worked here. It’s super simple, and basically does three things. First, it asks if you really want to go through with this. Second, it launches Sync Toy and syncs the files. Third it waits one minute for Dropbox to upload and then shuts down the computer.
We leave a shortcut on the desktop labeled “Shutdown PC,” and have the guys fire that to shut down. For those that are interested, here’s what is included in the .bat file. Feel free to use and mod it as needed if you like.
@ECHO OFF
:choice
ECHO Are you sure you want to sync the SD-8 files and shutdown the PC
set /P c=2024
if /I “%c%” EQU “Y” goto :shutdown
if /I “%c%” EQU “N” goto :abort
goto :choice
:shutdown
“C:\Program Files\SyncToy 2.1\SyncToyCmd.exe” -R
cls
ECHO Commencing shutdown in 60 seconds.
shutdown /s /t 60
pause
:abort@ECHO OFF
exit
When this runs, it simply launches Sync Toy and runs whatever sync folders we have set up. Within a few seconds, I can see Dropbox notifications on other computers that the files are updating. Cool.
Backup Hardware
One thing we noticed during that crash was that if the Mac Mini (running Win 7) goes down, we’re kind of dead in the water. Although, since it’s Hog PC, we just need another PC.
So, I configured my laptop, which already had Win7 installed in VMWare, with the Hog software, and linked it to the Dropbox folder. So now, if the Mac dies, we grab my laptop, fire up VMWare and in a few minutes, we’re back in action.
Maybe we’re limited in our ability to build big shows due to the monitor limitations, but we can run the weekend. But what if we loose our DMX widget?
Backup, Backup Hardware
We’d actually have to loose both widgets because we have two, but what if we did? We still have the ETC Paradigm system in-house, with a really cool touch screen in the booth.
So one of our projects this summer will be to update the control programming on that system to include all the moving lights and LEDs, as well as a simple 10-cue page that would get us through a service in case everything else failed.
So that’s our system right now. In the last few weeks, I’ve been checking to see that the backups are actually working, and being pushed up to Dropbox.
I now feel like I have one fewer thing to keep me awake on Saturday night.
Mike Sessler is the Technical Director at Coast Hills Community Church in Aliso Viejo, CA. He has been involved in live production for over 20 years and is the author of the blog, Church Tech Arts . He also hosts a weekly podcast called Church Tech Weekly on the TechArtsNetwork.
