Posted on: March 11th, 2010 by Famous Phil

Today, I will pursue a topic that all of us wish wasn’t necessary to discuss, but it is a fact of life. Backups are required in life and this isn’t a new concept, but poor backups can be devastating when you find out that it doesn’t work (and that is usually when you need it the most). I’ve spent a lot of time making changes to my backup systems over the years, but they are never perfect and I constantly find new flaws that I constantly work to fix.

A few weeks ago, I found a major flaw in my backup system that I never really thought about in the past.  My backups would run every night by essentially pushing the latest backup to another server that I run.  This server had an account on it where the backup would neatly upload then not be touched again until the next backup cycle ran.  To access the backup, all I had to do is login as root on my production system (the system that runs famousphil), then request the backup.  Passwords are entirely controlled through keys, so I didn’t have to do anything special or know anything to gain access to my backup server.  Unfortunately, if all I needed was my root password / key to do that, if a hacker got into my system, they essentially could completely destroy my backup system as well.

With the ever growing threat of hackers attempting to compromise my network of servers, I’ve began giving more thought to hack attempts and prevention along with hard disk and other physical server failures (which also happen).  Because of these thoughts, I’ve implemented a new backup routine that fixes most of my issues with a hacker gaining access to my network.  If I discover a hacker has penetrated one of my servers within 12 hours, I will always be able to restore a backup that is no more than 3 days old now.  The same goes for a hard disk failure / natural disaster. I do not feel it is in the best interest of Matthouse to disclose the new backup system and exact scheduled times, but I thought that it would be appropriate to share a little insight into that thought that I’m sure many other system administrators might fall into at one point or another.

Another threat that I found with my backup system was that part of my backups were incomplete.  For any admin that thinks backing up /var/lib/mysql is appropriate to backup MySQL, think again.  About 2 months ago, I thought this was appropriate, so I began backing up MySQL this way (I never really thought to check it initially since the server was brand new).  I normally check the redundancy of my backups about once every 2-3 months.  During my redundancy check, I always check everything and I discovered that I could not restore the MySQL database server from that directory easily.  Since I have the original server still running, I came up with a new backup procedure that works much better and can actually be restored.

For all of you who need to backup an entire MySQL server, here are the 2 commands that will help:

• Backup: mysqldump -u root -pPASSWORD –all-databases > backup.sql
• Restore: mysql -u root -pPASSWORD < backup.sql

Those are the 2 major points I wanted to make in this post.  In conclusion, I recommend that you make a backup schedule for your servers and verify that loop holes don’t exist in it.  I’m constantly looking for loop holes and I patch them as I find them.  Also, verify your backups!  I would have never realized that MySQL wasn’t being backed up unless it was for verifying that my backups are in tact and can be restored.  I verify my backups every other month and I recommend that every system admin does so.

For anyone at home, HARD DRIVES DO FAIL.  I strongly recommend keeping any data that you could not afford losing on a flash drive or external hard drive where if your computer did fail, you would still have your data.  Unfortunately, data is unrecoverable at cheap rates, making a backup is a cheap insurance policy to avoid complete disaster!

Tags: backup, hacker, hard drive failure, mysql
Posted in Hosting / Server Administration
|| 2 Comments »