Posted on: August 20th, 2010 by Famous Phil

This is an annoyance that I constantly have to look at 2 sources to remember so I’ve decided to consolidate my 2 sources into a single blog that I can refer to in the future. Basically, every time I set up a new Server 2008 R2 box to act as an Active Directory domain controller, I always run into passwords having to be complex and changed every 42 days by default. This is an annoyance because users don’t like having to do that so frequently. The location to change this isn’t in a common sense location either.

To change the policy, goto your start menu and in the search area (provided it will run commands), type in “gpmc.msc” and hit enter.  This will bring up the Group Policy Management Console.  Expand the local forest that you want to modify the password policy for.  Expand Domains, and the domain that you wish to modify.  Right click on Default Domain Policy and click edit.  This will bring up a new screen.  You want to navigate to Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Account Policies -> Password Policy.  From this screen, you will be able to modify the password policy.  You can either reboot or type in a new command “gpupdate /force” and the new settings will take over.

I don’t recommend disabling features such as complexity because your users will then be able to use passwords such as “letmein, password, changeme”, etc.  These are insecure passwords and those who use them are simply asking to get their accounts compromised.

Also, the solution above will not work if you have a password window open that is forcing you to use a complex window, you will have to go through that wizard again.  This also won’t reverse the force password changes on next login if a user needs to change their password due to the old policies.

Tags: active directory, domain controller, password complexity, password policy, server 2008 r2
Posted in Hosting / Server Administration
|| No Comments »

Posted on: August 6th, 2010 by Famous Phil

Often times, I’m asked the question: Should I get Windows hosting or Linux hosting for my new website.  This is a topic that comes up time and time again in my field of expertise and sadly, newbies often choose based on perception, not based on fact.  This blog will cover some of the facts about both hosting platforms and hopefully help you make an informed decision about how you want your website hosted.
Read the rest of this entry »

Tags: choice, hosting, Linux, windows
Posted in Hosting / Server Administration, Programming, Technology
|| 16 Comments »

Posted on: August 2nd, 2010 by Famous Phil

Introduction:

This topic plagues me to death every time I need to do some administrative function MySQL simply because I don’t do it every day.  I have 3 servers that I manage entirely via the command line now and all 3 require me to know at least some MySQL.  Unfortunately, I always end up going to several sources to get all the information I need.  So instead of doing that in the future, I’m writing this blog as a centralized reference for everything I need.  Hopefully you can use this blog as much as I will!

Note: You will need to click the “show code” icon in the top right corner to view the code entirely for some of the blocks that are longer than the code box.

Read the rest of this entry »

Tags: administration, cheat, command line, interpreter, mysql
Posted in Hosting / Server Administration, Programming
|| 4 Comments »

Posted on: July 14th, 2010 by Famous Phil

This is the revised version of my initial blog last night:

I always perform updates monthly on all of my servers beginning at 10:30pm Eastern Time of the Wednesday following the 2nd Tuesday of the month (to stay in line with Microsoft Updates).  I know a lot of Windows admins look at the 2nd Tuesday of each month as “Black Tuesday”, and I now have a first hand incident that has me dreading it also.

So last night, Windows Updates recommended I install Exchange 2010 Server Update Rollup 4.  I have never had issues in the past, so I quickly looked into known issues and guidance.  Nothing was listed with a Bing / Google search (I use Bing more when I’m dealing with Microsoft technologies).  Anyways, last night the updates took about an hour to install.  Once done, I rebooted as normal, but Outlook Web Access didn’t ever come back.  After some investigation, all of the exchange related services that make exchange work were disabled.  I’m not sure why this was, but I began troubleshooting lots of stuff and the I have 2 theories for what happened (I did both at the same time so it could be either one).

I first tried to remove the update rollup that Windows Update installed but was asked for a DVD that I don’t think was ever made and released by Microsoft!!!!  I also tried a system restore which lead nowhere since it doesn’t exist on Server 2008.  After this, I was considering my options, I tried 2 things at the same time which worked to fix the issue and get Update Rollup 4 installed (thankfully).

Theory 1: The updater package disabled the web service and since the web service had to be up to be updated, the updater failed to update everything successfully.

Theory 2: The previous update rollups (I had update rollups 1, 2, and 3 all installed) were interfering with the new update rollup.  So I removed all 3 previous ones then reinstalled update rollup 4.  Note that I had to uninstall all 3 of these with the broken install of rollup 4 still listed in the optional remove panel. Once I was ready to reinstall update rollup 4, only update 4 was listed in the installed updates for exchange section of Windows Updates.

Steps to fix such an error:

1. don’t panic or get impatient, this will take about 3 hours

First remove any previous update rollups from the add/remove programs in the control panel for applied updates.  Each one will take about 15 to 30 minutes to uninstall.  I started with rollup 3 and worked my way backwards.

2. get the official rollup 4 package from Microsoft’s website

3. open up an elevated (run as administrator) command prompt and change directory to the file that you downloaded

4. execute the rollup msp file – don’t do anything once the installer loads yet

5. open up the services console (under administrative tools)

6. start the updater… as soon as it is done with the stopping services text, immediately goto the services console and enable both the iis admin service and world wide web publishing service, also START them immediately.  There is about a 60 second window to do both (start the www publish service first)

7. let it install, afterwards you will need to re-enable all of the exchange related services that were running prior to the original windows update (I hope you have a good memory, for my server, it was all but the edgesync service)

8. reboot the server

9. hopefully exchange will work again, for me it did.  Apparently Microsoft released a bad update to Windows Update this past black Tuesday and it caught me off guard

Anyways, I hope this information is useful for other Windows Admins that are in this situation without any option but restore a complete system image and have lots of downtime while Exchange’s latest email is restored.

Posted in Hosting / Server Administration
|| 8 Comments »

Posted on: June 14th, 2010 by Famous Phil

When I build servers, I build them to run smoothly and last a long time.  I always keep security at the top of my priority list when configuring new servers (and even personal computers).  Today, my security practices were tested, and overall, were very successful!  As a few of my readers might know, Podnet, the IRC network that I run was exploited because of the IRC daemon (software) that it runs.  This required emergency maintenance that caused an hour outage today unexpectedly.

Before I go into my security and how I discovered the vulnerability I want to take a moment to defend open source software.  On June 12, Unrealircd’s developers discovered that the latest version of their software was hacked in November of 2009 on all the official mirrors and their main website.  This hacked copy was distributed for well over 7 months before someone finally caught on when their server was hacked.  When the first server was exploited, the developers figured out how very quickly and discovered that the official installation archive was the culprit.  Because of this, they learned that providing md5 checksums is very important in ensuring the original installation archive is used in installing software.  I have also learned how to check these archives and I will likely always check them from now on after being a victim myself.

Although this vulnerability was solved within hours of its discovery, there are many people out there that will defend proprietary software like Microsoft Windows which is full of vulnerabilities.  The truth of the matter is, Microsoft would have responded just the same was as Unrealircd’s developers did.  The fact that Unrealircd is open source truly allows anyone to see the source and see what vulnerabilities are in a program (and hence fix them).  Its sad that no one who downloads open source software from Unreal discovered this (me included) but that is the way it is.  I still support both Open Source and Proprietary software for their own unique reasons.

So now onto my security.  All of my systems run most programs as underprivileged users that cannot do any wide system damage.  I also run software that detects changes in any kind of file on my servers.  It is a combination of these 2 methods of prevention that prevented podnet from being down much longer than it was (due to the need for a complete server restore to last week).  Thankfully, the script that was exploited couldn’t do anything but run and immediately get killed sending me a notification that something was started remotely.  Within 10 minutes of the exploit, I began figuring out how to fix the problem.  I was originally going to wait until around now to fix it (when no one is on the network) but then within an hour another attempt happened.  Obviously I didn’t want to risk letting my security fail.

If nothing else, I hope that you got that you should NEVER EVER EVER run your own system as an administrator if you care about the integrity of your computer or system.  Thankfully, running as another user, I ended up replacing a single directory as a precaution on the server instead of doing an entire system restore that would have taken much longer.  I continue to always run as the user phil for my own systems which has no privileges except for what I need.   I also disable / rename the main administrator account since most hack attempts come to those accounts (since they’re default).  Hopefully you do the same

Tags: exploit, unrealircd, vulnerability
Posted in Hosting / Server Administration
|| 1 Comment »

Posted on: May 30th, 2010 by Famous Phil

As per a suggestion from John, I should take up local development.  For all of you who are lost, I mean development of websites on my own computer without any interaction with the outside world.  For years, I have developed on the actual server that will be hosting the website.  There are several pros / cons to both methods so I thought I’d outline a few of them in a blog post (it has been coming, I have been very busy as of late).  I also thought I’d outline a few problems with local development and solutions for them.

For years, Remote development has been my choice for every job I’ve been contracted to do.  My reasoning is simple, if I have a solid connection to the remote environment, I can develop on it and it is guaranteed to work once I’m finished.  After all, it is on the remote system where the site will eventually be permanently settled.  Unfortunately, with remote development, I usually have to be careful about not removing stuff that relates to the current system in place.  This is very true of website development, I wouldn’t want to remove the website that is currently in place and working.

For my last major project, retreadproducts.com (it has yet to be released), I chose to try local development.  Since I’ve recently adopted Ubuntu Linux as my operating system of choice (that is another blog post eventually), it would be very simple to mimic the remote server environment to minimize compatibility.  I must admit that local development was much faster and easier to do than remote development ever was.  It is because instead of a lag of a couple seconds (for remote development) to do any kind of update (plus upload time for the updated portions), I’ve discovered that I waste about half the project time just keeping a good connection to the Internet and waiting for files to go back and forth to the remote server.  This is a huge loss of time and in my humble opinion, I don’t think that it is worth developing on the remote system as my main development method.  I also found that mimicking the remote environment almost exactly made compatibility 100%.

Getting a local development environment setup is a task in itself, but with Ubuntu, it is simple as opening up a command line, typing in sudo tasksel, then selecting the webserver role.  Everything is setup for you practically, except for the configuration files… but then again, even those have a default configuration to make development easy.  For me, I had to tweak my configuration files a little bit to mimic the remote environment as much as possible.

During my tweaking I ran into a few issues, that for me weren’t a huge deal, being a big time CentOS administrator.  My first issue was very easy to fix… Ubuntu uses apache2 as its apache webserver service.  This kept throwing me for a loop since CentOS / RedHat both use httpd as the webserver service (or daemon for the Linux geeks out there).  Getting over this hump left one other major problem that I needed to solve… and that was the .htaccess files weren’t being used by the apache webserver.

In Linux webserver talk, .htaccess files are hidden files that provide advanced functionality for any website that they are in the same directory with.  I use .htaccess files mostly for rewriting a page like: http://famousphil.com/blog/index.php into http://famousphil.com/blog .  This is very powerful since without the .php extension, hackers don’t know what technology is serving your website.  Of course they can figure it out via other means, but .htaccess is very powerful at deterring script kiddies.  Naturally, I just gave out how this website is powered   It has been a while since I’ve configured a Windows IIS server, but I believe they have something similar for ASP pages.

It turns out that Ubuntu disables .htaccess by default.  Turning it on is very simple.  First off… you need to allow it in your apache configuration.

Step 1, view all the configuration files under /etc/apache2… especially apache2.conf… Make sure that AllowOverride is allowed, so as an example:

<Directory />
        Options FollowSymLinks
        AllowOverride all
</Directory>

Step 2:Enable the correct module in the Apache configuration:
Type in sudo a2enmod rewrite to a command line
then type in sudo /etc/init.d/apache2 restart

This will solve your issues with .htaccess not working, especially if you’re developing locally for wordpress like I was.

Tags: .htaccess, apache, local development, remote development
Posted in Hosting / Server Administration, Programming
|| No Comments »

Posted on: April 8th, 2010 by Famous Phil

Today I was reviewing my server logs and noticed the following error:

“ the working directory is not writable: 1 Time(s)”

The solution to this error can be found here: http://slaptijack.com/system-administration/dnsbind-issue-named-the-working-directory-is-not-writable/comment-page-1/#comment-19370

Here is a summary of the solution:

cd /var/named/chroot/var/
ls -la
If you see group not writable on the named directory like the following:
drwxr-x— 4 root named 4096 Jan  9 01:56 named/

issue the following commands:

chmod g+w named
rndc reload

The log file should no longer show this error.

Tags: bind, named, not writeable
Posted in Hosting / Server Administration
|| 2 Comments »

Posted on: April 5th, 2010 by Famous Phil

There comes a time in every admin’s career when knowing how to telnet is a necessary evil.  Telnet basically opens a network connection to a server on a specified port so that you can communicate with a server.  Telnet isn’t secure by any means, so this is an excellent tool for checking firewall blocks, email servers, web servers, etc.  I’m just going to outline a few tricks that I always find myself forgetting.

1. Escaping from Telnet: To escape from telnet, simply enter the key combination “ctrl + ]”.  This will bring up the telnet menu where you can then type in q for quit.  This will successfully disconnect a hung telnet session instead of having to take evasive measures like powering off the computer (I’ve done it).
2. Testing an Email Server:

Open a command prompt in either Windows, Mac, or Linux (chances are that Windows users will need to add it in the Add/Remove Windows Components area of the control panel).  Putty also supports telnet.

Telnet to the email server and port 25 (smtp) by “telnet email.server.com 25″.  SMTP is Simple Mail Transfer Protocol.

Type “HELO server.com.” or “EHLO server.com.”
Type “MAIL FROM:you@yourserver.com.”
You should see “250 ok”
Type “RCPT TO:friend@theirdomain.com.”
You should see “250 ok”
Type “DATA”
You might see — 354 Please start mail input.
Type “SUBJECT: Test Subject”
Type in your message as normal
Type “.” on the last line by itself

You might see 250 Mail queued for delivery meaning the mail will be sent.  At this point, you can exit telnet by typing in “quit” and enter.

Normally: this will allow for something called an open relay, and that is why computer scientists occasionally give SMTP the nickname “Stupid Mail Transfer Protocol”.  This was designed back in the 1970′s which is why this protocol is so vulnerable, unfortunately it is how Email functions.  There are better methods in the works, but compatibility prevents these from taking off.  For instance, Google Wave attempted to fix the problems in Email while making it much better.  Maybe it will take off some day because it is amazing, but right now, it doesn’t look good.

Open relays allow for anyone to connect to your mail server and send mail as someone else.  This is how spammers manage to spam so easily.  Because of this, many internet service providers block port 25 so that customers have to use the ISP’s heavily protected servers to send mail.  In addition, most providers make their servers only accept mail for valid users on that server.  This way, a spammer can’t send mail as anyone to someone that may be on another email system in the world.  This really helps cut down on spam.  For administrators, there is a relay testing service that is discussed in tip 3.

3. The last trick is testing that your mail server isn’t an open relay:  The easiest way of doing this is open up a telnet session to mail-abuse.org… so on your mail server’s terminal (command prompt), simply “telnet relay-test.mail-abuse.org”.  They will connect to your server remotely shortly after you connect and then return to you a full diagnostic list of tests and tell you about problems that they found.  This has to be done from the physical mail server, so you need to be the admin of that server to initiate this test.

Hopefully these telnet tricks might help you.  I know that I need them occasionally for my own system.

Tags: abuse, escape, mail, prevention, telnet
Posted in Hosting / Server Administration
|| 18 Comments »

Posted on: March 11th, 2010 by Famous Phil

For all of you who have no clue what single user mode is, it is a recovery environment for Linux based servers.  Single user mode allows for the system administrator account (root) to login via the local server (the console) to fix problems that prevent the server from coming up normally.

Single user mode can be triggered by a lot of factors, but there are two huge factors that can cause this.  One is when the system’s drive information file (fstab) becomes inconsistent with the server’s configuration.  The other time when a system will boot into this mode is when a file system is found to have errors on it.  Linux automatically checks its file system occasionally on boot, which is why single user mode may occur without much notice (that is if the check can’t quickly fix the problem on its own).

Single user mode presents only the root file system (/) to the root user after login and this file system has only read access.  This is to allow for file system checking since this mode is used commonly for running file system checks and repairs.  If your file system becomes corrupt, the last thing you want is for your computer to write to the file system and lead to further corruption.  The admin must mount the file system as read write in order to modify anything on the drive.

I’ve been in single user mode twice now for neglecting to verify all the information in the fstab file.  My first time in single user mode was not fun and I ended up using a Ubuntu live cd to fix the problem since I didn’t have enough patience to look up the command.  After my first time, I looked up the command in case I ever needed to enter single user mode again.  This effort paid off since I knew that I would be rebooting one of the servers a little earlier that had significant edits to the fstab file.  Although I was certain I didn’t make any mistakes, I didn’t check for duplicate entries.  This caused the server to boot into single user mode.  My job was to simply remove the duplicated line and reboot the server.  Since I can never remember the command, I decided to write this blog.  Remember this is a Centos 5.4 server running CPanel.

The command is: mount -n -o remount /

That command will mount the root file system (/) as a writable file system, thus allowing for a text editor such as vi to write edits back to the file system.  I simply used vi to remove the duplicated line in the /etc/fstab file and rebooted and the server is back up.  Hopefully I won’t see another time of single user mode, but as an admin, I’ve grown to expect stuff like this.  I thought I would share the command so I don’t ever get lost again trying to find it

Tags: mount, single user mode, write
Posted in Hosting / Server Administration
|| No Comments »

Posted on: March 11th, 2010 by Famous Phil

Today, I will pursue a topic that all of us wish wasn’t necessary to discuss, but it is a fact of life. Backups are required in life and this isn’t a new concept, but poor backups can be devastating when you find out that it doesn’t work (and that is usually when you need it the most). I’ve spent a lot of time making changes to my backup systems over the years, but they are never perfect and I constantly find new flaws that I constantly work to fix.

A few weeks ago, I found a major flaw in my backup system that I never really thought about in the past.  My backups would run every night by essentially pushing the latest backup to another server that I run.  This server had an account on it where the backup would neatly upload then not be touched again until the next backup cycle ran.  To access the backup, all I had to do is login as root on my production system (the system that runs famousphil), then request the backup.  Passwords are entirely controlled through keys, so I didn’t have to do anything special or know anything to gain access to my backup server.  Unfortunately, if all I needed was my root password / key to do that, if a hacker got into my system, they essentially could completely destroy my backup system as well.

With the ever growing threat of hackers attempting to compromise my network of servers, I’ve began giving more thought to hack attempts and prevention along with hard disk and other physical server failures (which also happen).  Because of these thoughts, I’ve implemented a new backup routine that fixes most of my issues with a hacker gaining access to my network.  If I discover a hacker has penetrated one of my servers within 12 hours, I will always be able to restore a backup that is no more than 3 days old now.  The same goes for a hard disk failure / natural disaster. I do not feel it is in the best interest of Matthouse to disclose the new backup system and exact scheduled times, but I thought that it would be appropriate to share a little insight into that thought that I’m sure many other system administrators might fall into at one point or another.

Another threat that I found with my backup system was that part of my backups were incomplete.  For any admin that thinks backing up /var/lib/mysql is appropriate to backup MySQL, think again.  About 2 months ago, I thought this was appropriate, so I began backing up MySQL this way (I never really thought to check it initially since the server was brand new).  I normally check the redundancy of my backups about once every 2-3 months.  During my redundancy check, I always check everything and I discovered that I could not restore the MySQL database server from that directory easily.  Since I have the original server still running, I came up with a new backup procedure that works much better and can actually be restored.

For all of you who need to backup an entire MySQL server, here are the 2 commands that will help:

• Backup: mysqldump -u root -pPASSWORD –all-databases > backup.sql
• Restore: mysql -u root -pPASSWORD < backup.sql

Those are the 2 major points I wanted to make in this post.  In conclusion, I recommend that you make a backup schedule for your servers and verify that loop holes don’t exist in it.  I’m constantly looking for loop holes and I patch them as I find them.  Also, verify your backups!  I would have never realized that MySQL wasn’t being backed up unless it was for verifying that my backups are in tact and can be restored.  I verify my backups every other month and I recommend that every system admin does so.

For anyone at home, HARD DRIVES DO FAIL.  I strongly recommend keeping any data that you could not afford losing on a flash drive or external hard drive where if your computer did fail, you would still have your data.  Unfortunately, data is unrecoverable at cheap rates, making a backup is a cheap insurance policy to avoid complete disaster!

Tags: backup, hacker, hard drive failure, mysql
Posted in Hosting / Server Administration
|| 2 Comments »