- Pip install Boto3 - Thu, Mar 24 2022
- Install Boto3 (AWS SDK for Python) in Visual Studio Code (VS Code) on Windows - Wed, Feb 23 2022
- Automatically mount an NVMe EBS volume in an EC2 Linux instance using fstab - Mon, Feb 21 2022
In my last post, I explained why I am strictly against scheduling backups. However, when it comes to restores I am a big fan of schedules. Another golden rule that belongs in every backup strategy is the backup of backups rule.
Schedule restores ^
Now before you check whether your backup software has a feature for scheduling restores, read on to see what I mean by "scheduling restores." Most backup experts recommend verifying regularly that backups are working properly. However, in my view, this is not enough. Just like manual backups are not a professional way to secure data, checking every now and then if the backed up data is really on the disks or tapes is certainly insufficient.
First of all, it is not very likely that you will really do it regularly. Be honest. When did you check your backups the last time? Second, it is not enough to check if the data was backed up. You really have to test the restored data in an environment that you prepared for this purpose.
The statistical data of failed restore attempts vary depending on who hired the analysts, but, I think, 50% is a reasonable working hypothesis. One thing is for sure—the main reason for failed restore attempts is because restores haven't been tested regularly.
Thus, my advice is to schedule a disaster recovery day in your IT department every two months, or at least every six months depending on the importance of your data. On this disaster recovery day, you can simulate a worst case scenario where a virus or a fire wipes out all your data. Try to restore your Active Directory, your Exchange servers, your databases, and user files in a virtual test environment. Just like a fire department needs regular training with real fires, an IT department needs to practice restoring data on real virtual servers. You will be surprised how you burn your fingers when you try this the first time.
Such a disaster recovery day has a nice side effect. Your management will realize that your IT department has more work to do than resetting user passwords. And if someone from management asks if this is really necessary, you can ask back if anyone has ever calculated the chances for the company to survive a full data loss because of a virus outbreak.
Offsite backups ^
Please don't think that I am paranoid, but backing up backups is indeed essential. Of course, I don't recommend installing a second backup solution that secures the data for your regular backup system. What I mean are offsite backups. While big companies have always done this by storing tapes at secure places far away from the company site, small organizations often neglect the risk of data loss as a result of a fire.
Offsite backups has become a new trend for two reasons. Firstly, the rise of CDP solutions has increased the likelihood that a virus not only destroys the data on all live systems but also the backups on storage systems. Erasing data in a tape library is much more difficult for malware than manipulating the disks on a server or storage system. Perhaps this is the main downside of CDP compared to traditional tape backups.
Secondly, new online backup solutions made offsite backups affordable for small companies. However, I don't recommend replacing your on-premises backup system with a cloud backup solution. I wouldn't hand over control of such a vital component of your company. But backing up backups in the cloud is an easy and affordable way to secure your backups from fires. And since special software is required to access the cloud storage, it is highly unlikely that a virus would be able to take this hurdle.
The good news is that CDP (contrary to tape backups) is perfectly suitable for offsite backups via online backups. Since your CDP solutions already collected all the data in your network at your central storage system, it is the ideal source for your second-level backup.
And just in case you worry about handing over all your organization's data to a third party, encryption technology can perfectly protect your data before it leaves your company site. You can use the encryption feature of the cloud backup software, or you can use your on-premises backup software to encrypt your data before the cloud backup software touches it.
Subscribe to 4sysops newsletter!
What about using tapes for offsite backups? To be honest, in my opinion, working with tape backups is another no-no. I had so many troubles in my IT career with the reliability of backup tapes and tape libraries that I now think that tape backups are not even a good solution for offsite backups. But I know that this point is highly disputed, and so I had better wait another year or two until "Don't use tape backups" becomes my fifth golden backup rule.