Live backup of a virtual machine with Virtual Server 2005 R2 SP1 using Volume Shadow Copy Service (VSS)
By Michael Pietroforte | 34 Comments | Permalink | Trackback | Previous | NextWhen I read recently that Virtual Server 2005 R2 SP1 will support Volume Shadow Copy Service (VSS) I was quite excited. This means that you can backup a running virtual machine without shutdowns. Some weeks ago I decided against Virtual Server 2005 and for VMware Server. However, VSS support is a killer feature in my view. Today, I tried to backup a running virtual Windows Server Virtual Server 2005 R2 SP1 Beta2 using Symantec Backup Exec 10d.
Of course you can backup a virtual server like any other server by installing the backup agent on the guest system. The charm of backing up a virtual server on the host system is that disaster recovery is very fast. If your host system has hardware problems, you restore all virtual servers within minutes on another host with completely different hardware.
So I was quite curious to learn more about the VSS support of Virtual Server. I used the Backup Exec Advanced Open File Option which also supports VSS. I backed up a running virtual Windows Server 2003 system without any error messages.
I then deleted the virtual server and restored it using Backup Exec. The restored Windows Server booted without any problems. However, when I logged on, I got a message informing me that the system was shut down unexpectedly. The Windows Server worked perfectly, though.
But it is obvious that this is not the way it is supposed to work. The main problem with this method is that you can save the virtual machine files (.vmc .vhd . .vsv) of a running virtual server, but you can’t save the system memory this way. So you might end up with a backup where the virtual machine files are not in a consistent state.
It is not enough if the backup software just supports VSS. It also has to support the new Virtual Server VSS Writer service (VS Writer). This VS Writer can be used by VSS requestors to back up and restore virtual machines. That way, backup software can inform Virtual Server that is going to backup a virtual machine and Virtual Server can create a snapshot before the copy process starts.
This is how the documentation of Virtual Server 2005 R2 SP1 describes the procedure:
In the console tree of the backup program, browse to Microsoft Virtual Server.
Expand Virtual Server.
Select the virtual machines that you want to archive.
Select the Virtual Server configuration for archive.
Select the backup type.
Select the media to which you want to back up the data.
Start the backup.
So you don’t select the virtual machine files, but the virtual machine as a whole.
Symantec doesn’t yet offer such an agent for Backup Exec supporting Virtual Server. I googled a little to see if there is any other backup software vendor offering an Virtual Server add-on already, but I didn’t find any. I guess it won’t take too long until the major vendors will have such products as soon as the final of Virtual Server 2005 R2 SP1 is available. Considering the description in the Virtual Server manual, it seems as if there is at least one working together with Microsoft since NTBackup won’t support it.
I am curious if VMware will offer something comparable in the new future. I still prefer VMware Server for many reasons. However, the possibility of being able to backup running virtual servers is a killer feature. Because of its high price, VMware ESX Server is not yet an option for us.
Leave a Comment | 
Subscribe RSS
| 
Newsletter
Great entry here, I have been searching all over for how to do a painless live backup of a virtual server. ESX is great, but way to expensive for SMBs, so I’m loving virtual server, except the host backup part which has been confusing me since i keep reading about the beta 2 VSS support and such. This helped clarify it for me.
VSS for Virtual Server is cool, i have been looking for documentation for it, i want to creat a script that will thell VS that a backup will be starting the get the script to copy the files to a NAS or somthing to do the backup that way…cannot find anything on that at the moment, has any one else got the api for the new feature in VS 2005 R2 SP1
The documentation you mention says “In the console tree of the backup program”… But since such a program doesn’t exist yet, this seems to be a placeholder only?
We actually don’t have at this time any program that does this, right?
You’re right this is a placeholder. Microsoft’s Data Protection Manager (DPM) 2007 will most likely support online backups of Virtual Server VMs. I’m not sure if this feature is already included in the beta. It is also possible that third party backup software vendors offer VSS support for Virtual Server meanwhile.
Yep, the data protection manager beta 2 will solve this problem.
I have made available a scriptlet for live virtual machine backup. It is especially designed for production environment. Available as free download from http://forum.hostbreak.com/showthread.php?t=302.
Aziz
Nice one, cheers for the script,
do not know if you all know, but DPM Beta 2 has been released on MS Connect and supports live backup of VM’s
Thanks – I was not aware of DPM. Thanks for the tip. I will look into it..
Aziz
Try this script :
http://cwashington.netreach.net/depo/view.asp?Index=1156&ScriptType=vbscript
The script looks nice… But “save state” means that VM still stop, even if it’s for a short time – probably at least five minutes…
Best Practice for Backing up Virtual Machines
After reviewing a brief article which asks more questions that it answers over at techtarget. “The best way to back up a virtual machine” I really thought about how a medium to large company does backups on their physical network and realiz…
This should be interesting:
http://redmondmag.com/columns/print.asp?EditorialsID=2324
Serge, thanks! That is indeed interesting. I might blog about this topic soon again.
http://redmondmag.com/columns/print.asp?EditorialsID=2324
This doesn’t work. It invokes a shadow copy, but I’ve been testing the script the last few days and all of the backups are identical file size and even dates.
If I restore to one of my backups, it acts as if its crashed and I don’t believe its transactionally consistent. In other words, back to the drawing board.
Bullet, I found otherwise in my testing. I started a virtual server (guest was win2k3 standard server) and wrote a text file down, then immediately fired the script. i then took the shadow copied files to another VS and loaded them up and it booted just fine and my test file was there an intact.
I didn’t compare file sizes, but the fact my changes I had just made were intact was good news. Curious what others find.
Don’t worry about file sizes, what about time and date stamps on the VHD and VSV files?
Are you certain that the virtual machine isn’t first being suspended and then restored? I’ve seen some cases where, even if not being specifically put to sleep, that this was happening. Usually with virtual machines that were originally created in earlier (pre SP1) versions of Virtual Server.
Try this: Start up some programs and leave them running. Perform the backup. When you restore to the backed up images are the programs still running?
If not, that (to me) is not a transactionally consistent backup, and unless someone proves otherwise I wouldn’t think that Exchange and SQL databases are being handled correctly.
I had done this when I did my test because I wasn’t sure of the state the copied files would be in when I started that VM up on another machine. Interesting enough, it was not in a saved state, so it was like the machine was just starting up, however as mentioned previousl my changed file was changed and the state of the VM during the shadow copy was not altered or suspended, so it was a live backup, somewhat consistent.
My conclusion was anything in memory and not written to disk will be discarded. This could have an impact on SQL and on Exchange. I guess it would be interesting to learn more about this script, I hope to use it in our production environment for backing up some of our virtual servers, would love to hear others thoughts.
I’ve been using scripts like this in production for about a year. The difference being that they suspend the virtual machine, take the snapshot, and then resume it. There’s anywhere from 30 seconds to 5 minutes of downtime depending on the size of the VM.
If I ended up with a customer that ran 24/7, I guess that would be a reason to use Data Protection Manager. Or just do the backup during shift change.
The script at redmondmag does not suspend the virutal server, it relies on the VSS writers which is why I am excited to use it. Are you saying that you used it and saw the virtual machine change states while it was running?
I’m not using the script at Redmond Mag. I tried it, and my own testing revealed that it simply does not work as advertised in the article.
The vshadow.exe utility doesn’t actually interface with Virtual Server. It just snapshots the disk volume on the host. In Microsoft’s own documentation, they say that the backup solution must be Virtual Server aware in order to work (as indicated in the article above), and vshadow isn’t.
The script _I_ use is a slight modification of this one:
http://cwashington.netreach.net/depo/view.asp?Index=1156&ScriptType=vbscript
There’s a less complex version of the above script linked in the Redmond Mag article (here if you are interested: http://cwashington.netreach.net/depo/view.asp?Index=1049&ScriptType=vbscript). If you look at the script Chris Wolf wrote, its basically the same script with the Virtual Server suspend/restore logic stripped out.
Thanks for the additional links, will scope these out!
One of the shortcomings of the various published scripts is that they only support snapping a single volume at a time. If your VM’s have VHD’s stored on multiple volumes (as ours do) then I think you’ll find you need to snap all the relevant volumes simultaneously eg:
vshadow.exe -p -script=setvar1.cmd %Vol1% %Vol2% %Vol3%
I haven’t had time to fully test this theory yet, but I wonder whether this may explain why some people have had problems running the scripts.
Bullet, I’ve done more testing with this redmon script and I’m very pleased. Here is what I did:
1) With my already running virtual server, I went to SQL Server, created a new table, inserted a couple rows into it.
2) immediately after this I went to start the redmonmag script on this virtual machine. The virtual machine was up 100% while this was running in the background.
3) when the script was done, I copied the virtual machine files over to another machine.
4) I did an add on the second virtual server to add these backed up files.
5) started it up, it was not in a saved state, it booted up and asked me for a reason for the unexpected previous shutdown.
6) I logged into SQL and my table was there and so were my records.
So All looks good in this scenario. I guess the DML and DDL I issued get written down to the SQL Log file immediately..I wasn’t sure if it took a while for the disk changes to make it down to the VHD, but at any rate, all of my transactions that were committed at the time of the script execution made it to copied server. So far I can’t find any negatives for this script. I wonder if what the guest OS is makes a difference? I’m running W2k3 Server.
5) was exactly the behaviour that was scaring me out of using the RedmondMag script. Your testing indicates to me that this might be worth while, so I won’t completely abandon my efforts.
Thanks!
I ran the Redmond Mag script on a production server for a week, and the other thing that scared me was that EVERY VHD backup had the same date and time stamp as the original VHD. A virtual machine file’s data and time stamps only change when it is shutdown or saved. How can it be that data INSIDE the file is changing but the VHD isn’t? It boggles me!
I’m not sure how the file isn’t being triggered, but somehow the changes I made in all of my testing I have done so far have been ported over, i also observed the same date / time stamp as you have. Maybe the date / time stamps are only tickled when the file physical grows (or shrinks). I’d be interested to see if you yield any different results from any additional tests.
I love these discussions. On the one hand you have the IT Commando forging ahead proving time and time again that backing up a live Virtual Machine without suspending it first and all that happens is that Windows OS complains it wasn’t shutdown correctly as a flag wasn’t set to say that it was.
On the other hand you have the nervous “but it does’nt back up what is currently in memory” therfore is not consistent. I beg to differ as Symantec Netbackup, Backup exec, Brightstor blah,blah clients do not do this either. They capture the system state information which isn’t superior, a mere collection of necessary systems bits and bobs inc registry, iis, boot info, com and dcom stuff etc, etc
The point is a Commando IT engineer in a High availability environment, now has the ability to restore an entire machine in a couple of relatively hastle free minutes (When using disk staging) and without the complaints of VPN users that decide that midnight is the best time of day for them to work.
Suspending every server for a short period every night might suit some, BUT who can actually say that in a server farm of 20 + servers all suspend + backup + unsuspend in 5-20 minutes, a little hopeful I feel. Plus what about when the thing gets stuck as you have to answer questions to get it going again.
I am not fond of being lynched when I walk in in the morning as there is only 1st line IT Support where I work.
Although of Course the IT Commando is a dead man if the server being backed up generates so much IO that the Volumes become corrupted (I have never seen this though). I have seen mild issue which are resolved by chkntfs /f, as during the backup data does change which leaves the ntfs bitmap a little confused as to where some files went and where these new ones came from.
Hmm, IT Commando, that has a nice ring to it
I’m very satisfied with what I’ve seen of the script and am rolling it out in my testbed now and fairly certain it will make it to my production environment before much longer.
I also enjoy these discussions and collaboration…the synergies formed from the collective are invaluable!
My own testing supports your work Trucking Nerd. Here’s the problem though. If the state of these “hot backup virtual servers” is indeed the equivalent of pressing the reset button (as indicated by the requirement for giving a reason for the unexpected shutdown), and if booting from a VM saved in this manner results in failure 1 time in 20, that’s 1 time too many.
And Computer Mischief – the backup systems you mention aren’t anything close to the same as backing up virtual machines from the outside. You can’t even compare them.
Concur, have you seen any cases where you couldn’t boot the copied virtual server?
Not yet, but I’m trying. I _have_ had Virtual Servers have issues on boot after they were shutdown abrubtly either by host restarts due to power loss, or shutdown (reset or turn off) from the VS console. However, the difference with the backup script is that its invoking Shadow Copy first on each volume. We’ve already agreed internally that if I test bringing up virtual backed up servers 20 times (i.e. from 20 different saved machines, not 20 times on the same machine) with NO issues, then we’ll use it on production boxes.
I’d be interested to learn if one of them is not bootable. I may do the same test on my side and will report back if I find anything different.
We think we are having success here using Chris Wolf’s (Redmond Mag) script to execute VS Writer to create the snapshot. Question though…
In Chris’ script, he copies the VHD files from the Volume Shadow Copy snapshot to another location, and his backup job backs up that location. Is it okay to just backup from the Volume Shadow Copy snapshot drive directly, instead of first copying the files out of there? We tested that, and restores seem to work. But I’m not sure if there’s something we’re missing that’s a reason why we should copy it first. Thanks.
I’ve tried to run this script on a server running 2005 R2 SP1 and when it runs the actual shadow copy component of the script, I watched it as it saved the state of each VM that was running on the server, not exactly what its supposed to do from how I read the script and article… or is it?
My question is there anyway now that I’m running SP1 of Virtual Server to do a VSS copy in real time without the servers being “save stated”?
It was possible prior to SP1 (we did it using a 3rd party tool that did VSS copies) and while I understand the implications of it being like “pressing reset”, this was acceptable to us for an emergency backup/restore scenario as it allowed us to make a backup of the image without having to turn it off.
Thanks.