Posted on: March 31st, 2010 by Famous Phil

I’ve spent the last year thinking of ways that I could effectively make a decent April Fools joke.  Unfortunately, I am not a creative person in any aspect and I often end up using the ideas of others to make seemingly creative and innovative progress.  I’ve never let this stand in my way of progress and I never intend to let it.  So instead of doing a prank, I’m going to discuss some “fails” over the past year that I’ve found that I liked for one reason or another.

But first… A few pranks that I was tossing ideas around for but ultimately decided to let go of.  Maybe some year

• Modify famousphil.com to have a “make believe” virus that would appear to infect your browser / computer.  I’m sure that you’ve seen those virus ads on malicious sites, I was considering making one of these, but not nearly as destructive.  I decided against this because users would be required to click ok… and well most users tend to click ok without reading (in my experience).  I didn’t really want to infect or compromise systems with something thrown together in a few hours for something like today.
• Cause intentional intermittent downtime to famousphil.  Of course, with this, readers may become discouraged and never come back and that isn’t good.
• Post an article on a major subject related to server administration that is actually a huge lie with fake commands that actually cause more confusion / damage than it is worth.  Obviously, april fools shouldn’t cause harmful damage so this was out the door immediately.
• Finally: edit all of the servers I manage to make common commands / operations looks like major errors occurred (when they really didn’t).  I considered this, but I’m not in touch with my other staff members as often as I would like, so they might consider it a real emergency and wake me up early on my sleep in day!

So now onto some of the fails of the year that I really didn’t like when they occurred:

• Last June: LXAdmins owner commits suicide after a major web hosting company is hacked and over 100,000 websites are deleted instantly.  It is rumored that this is caused by the owner of this web hosting company stored his root passwords in his GMail account which had a weak password.  A Matthouse  Virtual Server was one of the 100,000 that was affected (although sufficient backups are in place)   More Info
• Exchange 2003 and 2007 Server: I began implementing an Exchange Email system for myself around June.  I remember that when I first implemented 2003, I really hated everything about it.  I then upgraded to 2007 and didn’t really like that, so now I am at 2010 and I really like 2010.  The failure is that I spent well over 100 hours of my valuable time figuring out what I was going to do about email.  I’m still learning a lot about the backend of Exchange.
• My backup system, in December, I looked at my backup routine with a critical lens.  This was obviously outdated and I also was not even backing up half of the data.  I have since found much better methods to backups and they are implemented in the production systems.
• My inability to mount a drive as read / write in single user mode causes a 4 hour server outage.  About a month ago, I rebooted the main web server that hosts 80% of the Matthouse client sites to test my work on modifying the underlying disk partition structure.  It turns out that I was missing a single flag that I blogged about.
• My inability to read the server ID before requesting a complete wipe by the datacenter.  This caused a 24 hour outage around late December, early January.  This fact still sort of embarrasses me to this day, but I doubt I will ever do this again!  Good thing I fixed my backups just a week before this.  Good news is that the backups work perfectly!
• Last December, I was grading exams for my job as a Teaching Assistant with a few other TAs.  I was asked to generate a key for the other graders since I wrote a section of problems.  I should know the answers to my own problems… but for some reason I made a slight mistake and that propagated onto a little over 60 exams before it was caught.  It took another hour to fix.
• Last Failure: Microsoft Windows Mobile 6.5.1 Daylight Savings Time option missing: I use a custom built version of this Windows version that omitted the daylight savings time.  Because time shifted ahead a few weeks ago, I actually missed an appointment to figure out what I need to do in terms of classes.  My phone shifted correctly, but the times in my calendar never adjusted themselves, so I ended up doing it manually after discovering I was exactly 1 hour late for my appointment.

Obviously, I’m not perfect, but I do find that I learn from my mistakes quickly.  Hope you enjoyed reading a few of my biggest failures over the past year.  My next blog is going to deal with sound wave analysis and the “Loudness Wars”, but I need a bit more research before I jump into the topic.

Tags: april fools, fail
Posted in Student Life
|| 2 Comments »

Posted on: March 11th, 2010 by Famous Phil

For all of you who have no clue what single user mode is, it is a recovery environment for Linux based servers.  Single user mode allows for the system administrator account (root) to login via the local server (the console) to fix problems that prevent the server from coming up normally.

Single user mode can be triggered by a lot of factors, but there are two huge factors that can cause this.  One is when the system’s drive information file (fstab) becomes inconsistent with the server’s configuration.  The other time when a system will boot into this mode is when a file system is found to have errors on it.  Linux automatically checks its file system occasionally on boot, which is why single user mode may occur without much notice (that is if the check can’t quickly fix the problem on its own).

Single user mode presents only the root file system (/) to the root user after login and this file system has only read access.  This is to allow for file system checking since this mode is used commonly for running file system checks and repairs.  If your file system becomes corrupt, the last thing you want is for your computer to write to the file system and lead to further corruption.  The admin must mount the file system as read write in order to modify anything on the drive.

I’ve been in single user mode twice now for neglecting to verify all the information in the fstab file.  My first time in single user mode was not fun and I ended up using a Ubuntu live cd to fix the problem since I didn’t have enough patience to look up the command.  After my first time, I looked up the command in case I ever needed to enter single user mode again.  This effort paid off since I knew that I would be rebooting one of the servers a little earlier that had significant edits to the fstab file.  Although I was certain I didn’t make any mistakes, I didn’t check for duplicate entries.  This caused the server to boot into single user mode.  My job was to simply remove the duplicated line and reboot the server.  Since I can never remember the command, I decided to write this blog.  Remember this is a Centos 5.4 server running CPanel.

The command is: mount -n -o remount /

That command will mount the root file system (/) as a writable file system, thus allowing for a text editor such as vi to write edits back to the file system.  I simply used vi to remove the duplicated line in the /etc/fstab file and rebooted and the server is back up.  Hopefully I won’t see another time of single user mode, but as an admin, I’ve grown to expect stuff like this.  I thought I would share the command so I don’t ever get lost again trying to find it

Tags: mount, single user mode, write
Posted in Hosting / Server Administration
|| 1 Comment »

Posted on: March 11th, 2010 by Famous Phil

Today, I will pursue a topic that all of us wish wasn’t necessary to discuss, but it is a fact of life. Backups are required in life and this isn’t a new concept, but poor backups can be devastating when you find out that it doesn’t work (and that is usually when you need it the most). I’ve spent a lot of time making changes to my backup systems over the years, but they are never perfect and I constantly find new flaws that I constantly work to fix.

A few weeks ago, I found a major flaw in my backup system that I never really thought about in the past.  My backups would run every night by essentially pushing the latest backup to another server that I run.  This server had an account on it where the backup would neatly upload then not be touched again until the next backup cycle ran.  To access the backup, all I had to do is login as root on my production system (the system that runs famousphil), then request the backup.  Passwords are entirely controlled through keys, so I didn’t have to do anything special or know anything to gain access to my backup server.  Unfortunately, if all I needed was my root password / key to do that, if a hacker got into my system, they essentially could completely destroy my backup system as well.

With the ever growing threat of hackers attempting to compromise my network of servers, I’ve began giving more thought to hack attempts and prevention along with hard disk and other physical server failures (which also happen).  Because of these thoughts, I’ve implemented a new backup routine that fixes most of my issues with a hacker gaining access to my network.  If I discover a hacker has penetrated one of my servers within 12 hours, I will always be able to restore a backup that is no more than 3 days old now.  The same goes for a hard disk failure / natural disaster. I do not feel it is in the best interest of Matthouse to disclose the new backup system and exact scheduled times, but I thought that it would be appropriate to share a little insight into that thought that I’m sure many other system administrators might fall into at one point or another.

Another threat that I found with my backup system was that part of my backups were incomplete.  For any admin that thinks backing up /var/lib/mysql is appropriate to backup MySQL, think again.  About 2 months ago, I thought this was appropriate, so I began backing up MySQL this way (I never really thought to check it initially since the server was brand new).  I normally check the redundancy of my backups about once every 2-3 months.  During my redundancy check, I always check everything and I discovered that I could not restore the MySQL database server from that directory easily.  Since I have the original server still running, I came up with a new backup procedure that works much better and can actually be restored.

For all of you who need to backup an entire MySQL server, here are the 2 commands that will help:

• Backup: mysqldump -u root -pPASSWORD –all-databases > backup.sql
• Restore: mysql -u root -pPASSWORD < backup.sql

Those are the 2 major points I wanted to make in this post.  In conclusion, I recommend that you make a backup schedule for your servers and verify that loop holes don’t exist in it.  I’m constantly looking for loop holes and I patch them as I find them.  Also, verify your backups!  I would have never realized that MySQL wasn’t being backed up unless it was for verifying that my backups are in tact and can be restored.  I verify my backups every other month and I recommend that every system admin does so.

For anyone at home, HARD DRIVES DO FAIL.  I strongly recommend keeping any data that you could not afford losing on a flash drive or external hard drive where if your computer did fail, you would still have your data.  Unfortunately, data is unrecoverable at cheap rates, making a backup is a cheap insurance policy to avoid complete disaster!

Tags: backup, hacker, hard drive failure, mysql
Posted in Hosting / Server Administration
|| 2 Comments »