Posted on: February 28th, 2010 by Famous Phil

As of late, I’ve been focusing many of my efforts into Windows more than on Linux.  I suppose my only reason is because my Windows administration knowledge is quickly becoming outdated, but it is also because I spend 99.9% of my time developing and administrating Linux based machines.  I am usually under the impression that Windows makes stuff so easy with the “next next next finish” principle, but Windows has just as many problems as Linux does, it just does not make them evident until you actually want to do something with Windows technology.  The only great thing about Windows is that normally, the fix is 99 times easier than the same type of fix on a Linux based system.

Over the past couple of months, I’ve been wanting to put together a Windows based web server for myself.  Its purpose was originally to keep my Outlook 2007 application on Exchange open (to update my website calendar / update my webmail RSS feed folder).  I’m beginning to run other stuff on the server (e.g. my instant messenger) so that I can have a better online presence (lets face reality, email isn’t as live as some of my friends would like).  Since I chose to run Windows Server 2003, I decided that it might be worth my time to enable Internet Information Services and work on developing some ASP based web applications just for the fun of it.  Of course, when the time is right, I will open up a few more accounts on the server through Matthouse for others who share my desire to learn ASP for fun.  For those of you who are wondering, I’m waiting for a good billing system to come along, I have one, I’m simply waiting on the developer to make a few changes.

Anyways, today I was playing around with the server and I enabled FTP publishing so that I could easily upload a few files to the server through my local computer.  Unfortunate for me, going to the windows components page under the control panel’s add/remove programs section to add FTP Publishing wasn’t enough.  Sure enough, the server did install FTP and made it available in the Internet Information Services Manager section of the computer management console.

After I deleted the default ftp site and made a new site with isolated user home directories, ftp didn’t work correctly.  Note that this server does not connect to nor run Windows Active Directory, so all the accounts are local.    Anyways, after some research, I found that under the ftp root, I needed to add a folder called “LocalUser” and then I had to add the username of the account under that directory.  To ensure that other users could not read files / data the directory, I changed permissions on the user folder to allow only that user and administrators access. At this point FTP began to sort of work!

I say sort of work because FTP only worked in active mode, meaning the Windows Firewall was getting in the way.  After spending a few minutes looking for a solution, I figured that I would try adding the internet information services executable to the exceptions list.  Sure enough, this fixed the passive mode problem.  The steps I took to fix the problem were: 1. open up the Windows Firewall control panel console under the Windows control panel.  I went to the exceptions tab and added a problem.  I browsed for additional programs and added “C:\WINDOWS\system32\inetsrv\inetinfo.exe” to the exceptions list.  After this was added, passive mode worked as expected.

Although Server 2003 is outdated by now, it is still a very stable system to do any experimental work on, which is why I choose to run it over a better and never version of Windows.   It is also much cheaper to run than a newer version of Windows like the current Exchange Server which is running Windows Server 2008 R2 Datacenter.

Tags: Firewall, FTP, IIS, Microsoft
Posted in Hosting / Server Administration
|| 2 Comments »

Posted on: February 27th, 2010 by Famous Phil

Today, the other admin of Matthouse Hosting came to me with a serious concern.  He was able to single handily bring down Matthouse and the entire infrastructure that I have spent countless hours on securing.  After learning a little about this attack he used, I was able to successfully bring down other major sites that run on Linux based servers, which makes me wonder how old this attack really is and how well known it is.   Naturally, it is a major security concern when a system administrator like myself finds out how easy it is to completely bring a network server to its knees, so I immediately dropped everything and began researching a solution to the problem.  In the process, I decided it would be beneficial to blog about what I’ve learned.

A typical DoS attack works by sending TCP/IP packets to the web server in massive quantities.  These packets of data clog the lines that run to the server and cause it to overload.  Think of a typical DoS attack as a couple hundred of people knocking at your front door at once asking something from you.  You will be naturally overwhelmed and give up entirely.  Servers are not any different when subjected to these attacks.  Although there are no real prevention tips for these kinds of attacks, there are special routers that can reroute these attacks to black holes causing the service to never go out entirely.  Depending on the network capacity, any server on that network may exhibit slow connectivity.   The datacenter that hosts Matthouse experiences huge DoS attacks about 5 to 6 times a year and during this time, some of our servers are entirely unreachable.

DoS attacks tend to be caused by lots of hackers going after a certain site / service that they don’t like.  For example, often on Internet Relay Chat (IRC), hackers get mad at other hackers and they use their networks of compromised computers to send packets to the service, bringing it down.  DoS is by far the hardest type of attack to prevent, and admins such as myself dread them.  Matthouse has some redundancy, but I doubt it could handle a large scale attack without at least some period of downtime, then again, many smaller networks couldn’t handle a large scale attack, so Matthouse isn’t alone.

Moving on, The Slowloris attack (located at: http://ha.ckers.org/slowloris/) is classified as a special kind of Denial of Service (DoS) attack on a web server.  This attack works by sending partial request packets to the web server to request data in return.  It sends fully constructed internet packets so it doesn’t clog the network, but the packets it sends are not fully constructed, and the web server thinks that data is missing so it requests a resend of the packet.  Normally the web browser will send a full request again and the server will complete the request.  Notice that the server never drops the initial connection for a period of time.

The idea of slowloris is to send incomplete packets at specified intervals to take up all the available slots of a web server, thus effectively making the server wait for that single client.  It is basically an attack that makes the server sit there and wait for a full request instead of freeing its resources to handle other people who may want a page from it.  Usually web servers only have enough open slots to fill up 512 requests or about there.  So if you can get a single computer to utilize all of these resource openings, you can block service to everyone else.  The cool part is that the server logs will not note anything suspicious because the requests are legitimate, just incomplete.  Once the attack is finished, the log file will show a bunch of 400 errors for bad requests.  When this attack is stopped, the server will return to normal within seconds because servers don’t wait a long time before giving up on the connection.

So what exactly is affected?  I’ve found that all of my Linux based servers which run the Apache web server are vulnerable to this attack.  To my surprise, both of the Microsoft Windows Servers running Internet Information Services (IIS) were NOT susceptible to this attack, likely because they wait for a complete request before responding to it.  I’ve also read that lighttpd isn’t affected in this type of attack.

Now for the fun part, the solution.  Justin beat me to the punch line, but his solution is very effective at protecting all of the servers that we manage at Matthouse and Amphosted.  Both of us run updated versions of CSF and LFD (http://configserver.com/).  With CSF, there was a very simple configuration directive that can be added to the configuration file that effectively blocks this attack. You should add the following to your configuration file for CSF (on the appropriate line) “PORTFLOOD = “80;tcp;20;5″”.  After testing the server for this vulnerability again, CSF effectively blocked the attacking server, thus fixing the vulnerability.

Hopefully you can protect your servers from what I have learned.  I’ve found several major websites that are vulnerable to this attack!

Tags: attack, denial of service, DoS, security vulnerability, slowloris
Posted in Hosting / Server Administration
|| 4 Comments »

Posted on: February 15th, 2010 by Famous Phil

I feel like starting this post on the topic of recycling.  If you are like me, you have heard of the 3 R’s.  For those of you who don’t know what I’m talking about, the 3 R’s are Reduce, Recycle, and Reuse.  A good portion of the news and current events in society encourage recycling and saving the environment.  For the most part, the world is moving towards saving the environment and the world as we know it, and that is great.

You may be asking what the heck does the movement to save the environment have to do with an IP title.  IP stands for Internet Protocol and the internet runs on this protocol (means of communication).  When using IP, your computer uses an address that makes it reachable by all other computers in the world.  Normally, you can think of IP as the United States Postal Service (USPS) and an IP Address as your mailing address or P.O. box.  The internet currently uses 2 major versions of IP, version 4 and version 6.  IP version 4 has been around since the dawn of the internet and can support a little over 4 billion computer addresses at once.  Due to special addresses within the IP version 4 range, only about 2 billion of these addresses are actually usable.  Due to this limitation, IP version 6 was born around 10 years ago to support a much bigger address system (it still has a limit, but its far bigger than anything could ever use).  The problem is, IP version 6 is going to be a pain to shift to because it will require everyone to switch at once.  Most system admins believe in not fixing something unless its broke, which could hinder this a lot.

Quite a few years back, there were a few ranges of ip addresses that were reserved to be private addresses.  This means that if you used a private address, no router on the internet would accept traffic from this range of addresses and forward it onto other networks.  This allowed for the usual wireless router at home with that all too familiar ip address 192.168.0.1 or something similar.  I’m sure if you’ve ever touched one of these, you would see that number somewhere in the documentation.  The idea of this is to put a bunch of computers behind a single internet ip version 4 address and allow all the computers to communicate with the internet without actually needing to be directly on the internet.  This process is called Network Address Translation (NAT) and all routers practice this.  the 192.168.0.x range allows for 255 computers to be connected simultaneously.  For bigger networks in need of more addresses, there is the 10.x.x.x where x is a variable number from 1-255.  This allows for many more computers, about 16 milion addresses to be a bit more precise.

Technically, using all 10.x.x.x network addresses on the internet, we could likely last a long time to come without requiring IP version 6. So, why don’t we use more private addressing?

I asked myself a very good question, and my answer is that I don’t know.  Obviously, we need to have public IP addresses to some degree to allow for websites, virtual private network gateways, and other services to the internet.  Putting internet accessible services behind NAT devices would knock out the entire internet.  On the other hand (and the main purpose of this post) is that there are several universities and companies and institutions that horde public IP addresses and do absolutely nothing with them.

A prime example of this hording can be seen here at the University at Buffalo.  UB owns a class B block of ip addresses 128.205.x.x and recently licensed another block of ip addresses in another range to support their continued growth.  Normally I wouldn’t be against this, but UB is in a tight position because the state government is failing at maintaining a good budget and IP addresses now cost a lot since there aren’t many left.  UB probably gets a volume discount, but for me, I pay $20/year for each public IP address that I am assigned and most of that cost goes directly to the naming corporation that handles all IP addresses.  This is a classic supply and demand problem.  Also, UB firewalls most of their IP addresses entirely, I can name no more than 20 computers on the UB network as a whole that I’ve accesses that absolutely need access to the internet (Web servers, etc).  The rest of the UB network computers are holding onto valuable public IP addresses that are 100% fire walled and inaccessible from the outside world.  Why does UB waste money and horde IP addresses that it doesn’t need?

Many other universities like Berkley, and Princeton have similar sizes of IP address blocks and I’m sure that they do the same basic stuff.  Now onto the ISPs.  Wildblue and Verizon both assign public unfirewalled IP addresses to their customers.  I can see why they might do this, but really, do customers need a public IP address just to potentially compromise the network and the internet computers around them?  Probably not.

I really feel that public IP addresses should only be handled by the major internet providers and datacenters that handle internet accessible servers.  Private companies that want real IP addresses should need to justify the use of those addresses.  If this was enacted, I’m sure that the internet addressing crisis would be a lot less of a problem.

Nat can definitely be the key to fixing the problem, but first, these big institutions that horde IP addresses need to give those addresses up and use them as they need them, just like many of the major hosting companies currently do.

Tags: address, crisis, internet, ip, running out, waste
Posted in Hosting / Server Administration, Technology
|| 13 Comments »

Posted on: February 2nd, 2010 by Famous Phil

This post is going to be short, sweet, and to the point   It seems that every time I build a new server, I’m constantly at a loss for memorizing the commands that generate public keys in SSH (Secure Shell).  SSH for the computer illiterate is the Linux equivalent to Windows Telnet Services.  This basically gives you a command prompt that is running on a remote computer.

SSH is the heart and soul of linux based computers because *everything* can be done through the command line simply.  Unfortunately, Windows is just starting to catch up with Telnet using a new application called powershell.  Powershell is very unlike the linux command line though since it is more a scripting language than a command based language.  This is why I always scratch my head when working with Microsoft Exchange Server’s Powershell commands and end up reverting to the famous Microsoft graphical administration interface which gives me the option of “next next next finish”.  I really do appreciate the Microsoft wizards that make software so easy to administer.  Anyways, Linux never started with a graphical desktop (Unlike Windows and Macintosh), therefore, the command line in linux is far easier to use when administering a system.

To get back to my original topic, I am a fan of passwordless entry into my own systems.  With linux and SSH, there are a few authentication methods, 1 being password entry, and 2 being public key authentication.  With a password login, you basically enter a username and password and you are logged into a command prompt on linux.

With public key authentication, you generate 2 keys at your local computer, one being a private key and the other being a public key.  Basically, you give the server you wish to connect to the public key, and only you hold onto the private key.  When you connect to the server, it will first give you the public key so that you can verify that the server is actually the server that you want to connect to.  This will be ignored on the first connect, but in the future, if this key changes, you will be notified that the server was possibly compromised.  Upon allowing the connection, you will give the server your private key and it will run some calculations on the key you provided.  If the result matches that of the server’s public key, you will be granted access.  Since keys are generally 256 hexadecimal characters long, these are much more secure than normal passwords, and they are generally much easier to use (unless your laptop is stolen).

So now onto how to actually use these keys:

1. Generate the key on your local machine (not the remote machine).
A. ssh-keygen -t rsa
2. Verify that your remote server has an ~/.ssh directory.  ~ in linux is the home directory.
3. Copy the local public key to the server
A. This can be done with: scp ~/.ssh/id_rsa.pub user@remote_server:~/.ssh/authorized_keys2
B. Notice I specified authorized_keys2… this is actually for SSH Protocol 2 which is a more secure SSH protocol than the original.
C. scp is secure copy in linux and it copies files between computers using SSH tunnels.
4. Attempt to ssh into your remote machine.  It should not need a password anymore.
A. ssh user@remote_server

Hopefully this helps make sense of SSH keys.

Afterthought on 2/4/10:
when using the scp command as a copy / paste above, this will overwrite your authorized_keys2 file on the remote computer.  To prevent this from happening, append to the file using this command (thanks Cris for pointing this out to me):

cat ~/.ssh/id_rsa.pub | user@remote_server “cat >> ~/.ssh/authorized_keys2“

Tags: key, keygen, Linux, ssh
Posted in Hosting / Server Administration
|| 6 Comments »